Compare commits

..

59 Commits

Author SHA1 Message Date
Krishnan Parthasarathi bbb56739bd Add User-Agent header with MinIO release details in http logs (#7843)
This would allow http log target server to distinguish between log
messages across different versions of MinIO deployments.
2019-08-14 11:43:43 -07:00
Nitish Tiwari 1cd801b2e9 Fix DeleteObjects() to remove renamed objects inside (#8072) 2019-08-14 11:15:25 -07:00
Aditya Manthramurthy bf9b619d86 Set the policy mapping for a user or group (#8036)
Add API to set policy mapping for a user or group

Contains a breaking Admin APIs change.

- Also enforce all applicable policies
- Removes the previous /set-user-policy API

 Bump up peerRESTVersion

Add get user info API to show groups of a user
2019-08-13 13:41:06 -07:00
kannappanr bc79b435a2 Doc: Fix PutObject API known limitations text (#7704) 2019-08-13 08:50:43 -07:00
maihde 0ed6daab59 fix: #8051 so that stale DNS entries are cleaned-up (#8053) 2019-08-13 08:49:26 -07:00
Praveen raj Mani e96f19c867 Use default ports for the schemes when dialing (#7960)
Fixes #7946
2019-08-13 16:43:11 +05:30
Harshavardhana aaef18b1a3 Fail for disks which overflow upon usage calculation (#8056)
Fixes #8035
2019-08-13 02:58:43 -07:00
Kaan Kabalak a48a034e5a Make directory path in the header editable (#8018)
This change will allow users to navigate to their desired locations,
including buckets and directories that haven't been "created" yet

Fixes #7883

Add tests

Change tooltip wording

Migrate to Font Awesome 5 to use path icon

Fix sidebar not closing on mobile
2019-08-12 22:36:19 -07:00
Harshavardhana bf8ec8ad73 Cleanup ui-errors and print proper error messages (#8068)
* Cleanup ui-errors and print proper error messages

Change HELP to HINT instead, handle more error
cases when starting up MinIO. One such is related
to #8048

* Apply suggestions from code review
2019-08-12 21:25:34 -07:00
Harshavardhana 8ce424bacd Enhance audit logging to capture responseTimes (#8067)
Audit logging requires to have

- timeToFirstByte
- timeToResponse

timing information
2019-08-12 20:32:34 -07:00
Anis Elleuch cea3e3f7a6 browser: Add user-agent header filter to gorilla mux route (#8040)
When a peer client which higher version sends a request to a peer
server with lower version, the returned status code is 200 OK instead
of 405 code. The reason is that the peer client request reaches the
browser handler, which registers itself by '/minio' route but without
any other constraints. Adding filtering by user agent header to the
browser route so internal requests to old endpoints versions return
405 error code.
2019-08-12 17:05:30 -07:00
Harshavardhana af36c92cab With ListBuckets() access-list only buckets the user has access (#8037)
This is a behavior change from AWS S3, but it is done with
better judgment on our end to allow the listing of buckets only
which user has access to.

The advantage is this declutters the UI for users and only
lists bucket which they have access to.

Precursor for this feature to be applicable is a policy
must have the following actions

```
s3:ListAllMyBuckets
```
and
```
s3:ListBucket
```

enabled in the policy.
2019-08-12 10:27:38 -07:00
Jakob Ackermann 1b258da108 [web-router] update the white list for favicons (#8024) 2019-08-11 22:17:02 -07:00
Andreas Auernhammer 35427a017d fix type conversion in UpdateKey for Vault (#8058)
This commit fixes a type conversion in the `UpdateKey`
implementation of Vault.
2019-08-11 22:20:25 +05:30
Cary FitzHugh a0715f3b59 Adding Notes on Swarm Network Aliases (#8054) 2019-08-10 21:25:01 +05:30
Harshavardhana 5a28ef0d47 Bump readiness check upto 10000 go-routines (#8057)
Most of our current workloads reach this value
regularly, it doesn't make sense to keep 1000
go-routine limit.
2019-08-10 18:13:14 +05:30
poornas 3385bf3da8 Rewrite cache implementation to cache only on GET (#7694)
Fixes #7458
Fixes #7573 
Fixes #7938 
Fixes #6934
Fixes #6265 
Fixes #6630 

This will allow the cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

- All cached content is automatically bitrot protected.

- Avoid ETag verification if a cache-control header
is set and the cached content is still valid.

- This PR changes the cache backend format, and all existing
content will be migrated to the new format. Until the data is
migrated completely, all content will be served from the backend.
2019-08-09 17:09:08 -07:00
Anis Elleuch 1ce8d2c476 Add bucket lifecycle expiry feature (#7834) 2019-08-09 10:02:41 -07:00
Harshavardhana a8296445ad Safely use unsafe.Pointer to avoid crashes on ARM (#8027)
Refactor the Dirent parsing code such that when we
calculate offsets are correct based on the platform
This PR fixes a silent potential crash on ARM
architecture.
2019-08-09 08:54:11 -07:00
Nitish Tiwari 43c72374d4 Fix healthcheck to look for entry of all IPv4 addresses (0.0.0.0) (#8046)
Fixes #7993
2019-08-09 01:28:37 -07:00
Harshavardhana d759a7ce99 Fix time formatting of Expiration field in STS (#8043)
Without explicit conversion to UTC() from Unix
time the zone information is lost, this leads
to XML marshallers marshaling the time into
a wrong format.

This PR fixes the compatibility issue with AWS STS
API by keeping Expiration format close to ISO8601
or RFC3339

Fixes #8041
2019-08-08 15:44:57 -07:00
Aditya Manthramurthy 5d2b5ee6a9 Refactor IAM to use new IAMStorageAPI (#7999) 2019-08-08 15:10:04 -07:00
Harshavardhana 9ee619ea14 Remove deprecated ENVs 2019-08-07 18:57:38 -07:00
Minio Trusted 55b385beee Update yaml files to latest version RELEASE.2019-08-07T01-59-21Z 2019-08-07 02:05:41 +00:00
kannappanr 930943f058 Fix IAM users migration regression in etcd (#8029)
PR #8008 did not migrate user data stored in etcd.
This PR fixes that.
2019-08-06 17:06:31 -07:00
Harshavardhana e6d8e272ce Use const slashSeparator instead of "/" everywhere (#8028) 2019-08-06 12:08:58 -07:00
Harshavardhana b52b90412b Avoid data-transfer in distributed locking (#8004) 2019-08-05 11:45:30 -07:00
Harshavardhana 843f481eb3 Allow "tmp" directory to be not available (#8021)
Also additionally add more context to the errors
generated by filesystem, to facilitate better
debugging.
2019-08-05 11:41:29 -07:00
Andreas Auernhammer f6d0645a3c fix DoS vulnerability in the content SHA-256 processing (#8026)
This commit fixes a DoS issue that is caused by an incorrect
SHA-256 content verification during STS requests.

Before that fix clients could write arbitrary many bytes
to the server memory. This commit fixes this by limiting the
request body size.
2019-08-05 10:06:40 -07:00
Aditya Manthramurthy 414a7eca83 Add IAM groups support (#7981)
This change adds admin APIs and IAM subsystem APIs to:

- add or remove members to a group (group addition and deletion is
  implicit on add and remove)

- enable/disable a group

- list and fetch group info
2019-08-02 14:25:00 -07:00
maihde 5cd9f10a02 Support Federation on a single machine (#8009)
When checking if federation is necessary, the code compares
the SRV record stored in etcd against the list of endpoints
that the MinIO server is exposing.  If there is an intersection
in this list the request is forwarded.

The SRV record includes both the host and the port, but the
intersection check previously only looked at the IP address.  This
would prevent federation from working in situations where the endpoint
IP is the same for multiple MinIO servers.  Some examples of where this
can occur are:
 - running mulitiple copies of MinIO on the same host
 - using multiple MinIO servers behind a NAT with port-forwarding
2019-08-02 12:40:51 -07:00
Praveen raj Mani b976521c83 Ignore faulty disks in xl-sets Storage info (#7878) 2019-08-02 12:17:26 -07:00
SCDealy 2c3b1f01d9 Update README.md (#8006)
Since MinIO by default is not fully S3 compatible, this fact should be
specified in a prominent place in the quick start guide so people 
new to MinIO don't have to spend hours figuring it out the hard way.
2019-08-02 10:31:07 +05:30
Andreas Auernhammer a6f4cf61f2 add UpdateKey method to KMS interface (#7974)
This commit adds a new method `UpdateKey` to the KMS
interface.

The purpose of `UpdateKey` is to re-wrap an encrypted
data key (the key generated & encrypted with a master key by e.g.
Vault).
For example, consider Vault with a master key ID: `master-key-1`
and an encrypted data key `E(dk)` for a particular object. The
data key `dk` has been generated randomly when the object was created.
Now, the KMS operator may "rotate" the master key `master-key-1`.
However, the KMS cannot forget the "old" value of that master key
since there is still an object that requires `dk`, and therefore,
the `D(E(dk))`.
With the `UpdateKey` method call MinIO can ask the KMS to decrypt
`E(dk)` with the old key (internally) and re-encrypted `dk` with
the new master key value: `E'(dk)`.

However, this operation only works for the same master key ID.
When rotating the data key (replacing it with a new one) then
we perform a `UnsealKey` operation with the 1st master key ID
and then a `GenerateKey` operation with the 2nd master key ID.

This commit also updates the KMS documentation and removes
the `encrypt` policy entry (we don't use `encrypt`) and
add a policy entry for `rewarp`.
2019-08-01 15:47:47 -07:00
Minio Trusted dfa8835720 Update yaml files to latest version RELEASE.2019-08-01T22-18-54Z 2019-08-01 22:27:41 +00:00
Anis Elleuch c5ac901e8d xl: Fix healing empty directories (#8013)
After some extensive refactors, it turned out empty directories
are not healed and heal status is also not reported correctly.

This commit fixes it and adds the appropriate unit tests
2019-08-01 14:13:06 -07:00
Aditya Manthramurthy 4101d4917c Fix IAM users migration regression (#8008) 2019-08-01 12:31:04 -07:00
maihde d966d29fed fix: add integer code for Windows Subsystem for Linux filesystem (#8010) 2019-08-01 06:00:57 -07:00
Minio Trusted c301f5882d Update yaml files to latest version RELEASE.2019-07-31T18-57-56Z 2019-07-31 19:06:20 +00:00
Harshavardhana 123cccaed1 Honor connection pooling while tracing (#7979)
This PR fixes relying on r.Context().Done()
by setting

```
Connection: "close"
```

HTTP Header, this has detrimental issues for
client side connection pooling. Since this
header explicitly tells clients to turn-off
connection pooling. This causing pro-active
connections to be closed leaving many conn's
in TIME_WAIT state. This can be observed with
`mc admin trace -a` when running distributed
setup.

This PR also fixes tracing filtering issue
when bucket names have `minio` as prefixes,
trace was erroneously ignoring them.
2019-07-31 11:08:39 -07:00
Anis Elleuch cbd02c58be federation: Avoid printing context canceled error (#7997)
Golang proactively prints this error
        `http: proxy error: context canceled`

when a request arrived to the current deployment and
redirected to another deployment in a federated setup.

Since this error can confuse users, this commit will
just hide it.
2019-07-31 11:08:10 -07:00
Aditya Manthramurthy c71895f225 Listen for PolicyDB events from etcd and fix etcd watch handling (#7992) 2019-07-30 18:50:49 -07:00
Harshavardhana b83413b167 Use GOPROXY to speed up builds (#7984)
Read more here https://proxy.golang.org proposal 
for go1.13
2019-07-30 22:27:11 +05:30
Praveen raj Mani 63e0a81760 Ignore stale notification queues in notification.xml (#7673)
Allow renaming/editing a notification config. By replying with 
a successful GetBucketNotification response, without checking 
for any missing config ARN in targetList.

Fixes #7650
2019-07-30 14:19:06 +05:30
Harshavardhana 8d47ef503c Fix crash observed in OPA initialization (#7990)
Related to #7982, this PR refactors the code
such that we validate the OPA or JWKS in a
common place.

This is also a refactor which is already done
in the new config migration change. Attempt
to avoid any network I/O during Unmarshal of
JSON from disk, instead do it later when
updating the in-memory data structure.
2019-07-29 15:58:25 -07:00
Harshavardhana 54eded2e6f Do not assume all HTTP errors as Network errors (#7983)
In situations such as when client uploading data,
prematurely disconnects from server such as pressing
ctrl-c before uploading all the data. Under this
situation in distributed setup we prematurely
disconnect disks causing a reconnect loop. This has
an adverse affect we end up leaving a lot of files
in temporary location which ideally should have been
cleaned up when Put() prematurely fails.

This is also a regression which got introduced in #7610
2019-07-29 14:48:18 -07:00
Harshavardhana 94c88890b8 Add additional logging for OPA connections (#7982) 2019-07-28 08:33:25 +05:30
Harshavardhana e871e27562 Refactor and simplify etcd helpers used in IAM subsystem (#7980) 2019-07-26 13:42:54 -07:00
Harshavardhana 007a52b546 Add common validation for compression and encryption (#7978) 2019-07-26 02:41:16 -07:00
Praveen raj Mani efb8b00db0 Preserve tailing backslash in URL paths (#7678)
Fixes #7649
2019-07-25 20:55:09 -07:00
Harshavardhana d744865dc6 Enable config for NAS gateway mode (#7948)
Starting with #7751 we don't store config
in etcd anymore, allow NAS to honor config
on disk.
2019-07-25 17:41:25 -07:00
Harshavardhana e40c29e834 Fail appropriately if the disk has I/O errors (#7972)
If the disk has I/O errors, we should simply ignore
such a disk and not be bothered about it - until
it is replaced.
2019-07-25 13:35:27 -07:00
Praveen raj Mani b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00
Harshavardhana 6f2b4675fa Add krb5 support for HDFS gateway (#7933) 2019-07-24 18:05:48 -07:00
Harshavardhana a4ce1daf99 docs: Use --user to start container in non-root (#7966) 2019-07-24 17:35:52 -07:00
Aditya Manthramurthy 7bdaf9bc50 Update on-disk storage format for users system (#7949)
- Policy mapping is now at `config/iam/policydb/users/myuser1.json`
  and includes version.

- User identity file is now versioned.

- Migrate old data to the new format.
2019-07-24 17:34:23 -07:00
Praveen raj Mani 55d4eee6f1 Enable event persistence in MySQL and PostgreSQL (#7629) 2019-07-24 10:18:29 -07:00
Harshavardhana ac82798d0a Remove uneeded calls on FS (#7967) 2019-07-24 15:59:13 +05:30
Minio Trusted 5b71c21330 Update yaml files to latest version RELEASE.2019-07-24T02-02-23Z 2019-07-24 02:09:19 +00:00
216 changed files with 6925 additions and 3874 deletions
+2
View File
@@ -26,6 +26,7 @@ matrix:
- ARCH=x86_64
- CGO_ENABLED=0
- GO111MODULE=on
- GOPROXY=https://proxy.golang.org
# Enable build cache
# https://restic.net/blog/2018-09-02/travis-build-cache
cache:
@@ -52,6 +53,7 @@ matrix:
- ARCH=x86_64
- CGO_ENABLED=0
- GO111MODULE=on
- GOPROXY=https://proxy.golang.org
go: 1.12.5
script:
- go build --ldflags="$(go run buildscripts/gen-ldflags.go)" -o %GOPATH%\bin\minio.exe
+1
View File
@@ -8,6 +8,7 @@ WORKDIR /go/src/github.com/minio/minio
RUN apt-get update && apt-get install -y jq
ENV GO111MODULE=on
ENV GOPROXY=https://proxy.golang.org
RUN git config --global http.cookiefile /gitcookie/.gitcookie
+15 -15
View File
@@ -30,35 +30,35 @@ verifiers: getdeps vet fmt lint staticcheck spelling
vet:
@echo "Running $@"
@GO111MODULE=on go vet github.com/minio/minio/...
@GOPROXY=https://proxy.golang.org GO111MODULE=on go vet github.com/minio/minio/...
fmt:
@echo "Running $@"
@GO111MODULE=on gofmt -d cmd/
@GO111MODULE=on gofmt -d pkg/
@GOPROXY=https://proxy.golang.org GO111MODULE=on gofmt -d cmd/
@GOPROXY=https://proxy.golang.org GO111MODULE=on gofmt -d pkg/
lint:
@echo "Running $@"
@GO111MODULE=on ${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/cmd/...
@GO111MODULE=on ${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/pkg/...
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/cmd/...
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/pkg/...
staticcheck:
@echo "Running $@"
@GO111MODULE=on ${GOPATH}/bin/staticcheck github.com/minio/minio/cmd/...
@GO111MODULE=on ${GOPATH}/bin/staticcheck github.com/minio/minio/pkg/...
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/staticcheck github.com/minio/minio/cmd/...
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/staticcheck github.com/minio/minio/pkg/...
spelling:
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find cmd/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find pkg/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find docs/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find buildscripts/`
@GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find dockerscripts/`
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find cmd/`
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find pkg/`
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find docs/`
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find buildscripts/`
@GOPROXY=https://proxy.golang.org GO111MODULE=on ${GOPATH}/bin/misspell -locale US -error `find dockerscripts/`
# Builds minio, runs the verifiers then runs the tests.
check: test
test: verifiers build
@echo "Running unit tests"
@GO111MODULE=on CGO_ENABLED=0 go test -tags kqueue ./... 1>/dev/null
@GOPROXY=https://proxy.golang.org GO111MODULE=on CGO_ENABLED=0 go test -tags kqueue ./... 1>/dev/null
verify: build
@echo "Verifying build"
@@ -71,8 +71,8 @@ coverage: build
# Builds minio locally.
build: checks
@echo "Building minio binary to './minio'"
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/minio 1>/dev/null
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/dockerscripts/healthcheck $(PWD)/dockerscripts/healthcheck.go 1>/dev/null
@GOPROXY=https://proxy.golang.org GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/minio 1>/dev/null
@GOPROXY=https://proxy.golang.org GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/dockerscripts/healthcheck $(PWD)/dockerscripts/healthcheck.go 1>/dev/null
docker: build
@docker build -t $(TAG) . -f Dockerfile.dev
+7 -2
View File
@@ -1,10 +1,15 @@
# MinIO Quickstart Guide
[![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
MinIO is an object storage server released under Apache License v2.0. It is compatible[1] with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
MinIO server is light enough to be bundled with the application stack, similar to NodeJS, Redis and MySQL.
[1]: MinIO in its default mode is faster and does not calculate MD5Sum unless passed by client. This may lead to incompatibility with few S3 clients like s3ql that heavily depend on MD5Sum. For full compatibility with Amazon S3 API, start MinIO with `--compat` option.
```sh
minio --compat server /data
```
## Docker Container
### Stable
```
@@ -86,7 +91,7 @@ service minio start
Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow [How to install Golang](https://golang.org/doc/install). Minimum version required is [go1.12](https://golang.org/dl/#stable)
```sh
GO111MODULE=on go get github.com/minio/minio
GOPROXY=https://proxy.golang.org GO111MODULE=on go get github.com/minio/minio
```
## Allow port access for Firewalls
+1 -1
View File
@@ -23,7 +23,7 @@
<!--[if lt IE 11]>
<div class="ie-warning">
<div class="iw-inner">
<i class="iwi-icon fa fa-warning"></i>
<i class="iwi-icon fas fa-exclamation-triangle"></i>
You are using Internet Explorer version 12.0 or lower. Due to security issues and lack of support for Web Standards it is highly recommended that you upgrade to a modern browser
+1 -1
View File
@@ -16,7 +16,7 @@
import "babel-polyfill"
import "./less/main.less"
import "font-awesome/css/font-awesome.css"
import "@fortawesome/fontawesome-free/css/all.css"
import "material-design-iconic-font/dist/css/material-design-iconic-font.min.css"
import React from "react"
+8 -8
View File
@@ -84,32 +84,32 @@ export class BrowserDropdown extends React.Component {
<li>
<Dropdown pullRight id="top-right-menu">
<Dropdown.Toggle noCaret>
<i className="fa fa-reorder" />
<i className="fas fa-bars" />
</Dropdown.Toggle>
<Dropdown.Menu className="dropdown-menu-right">
<li>
<a target="_blank" href="https://github.com/minio/minio">
GitHub <i className="fa fa-github" />
GitHub <i className="fab fa-github" />
</a>
</li>
<li>
<a href="" onClick={this.fullScreen}>
Fullscreen <i className="fa fa-expand" />
Fullscreen <i className="fas fa-expand" />
</a>
</li>
<li>
<a target="_blank" href="https://docs.min.io/">
Documentation <i className="fa fa-book" />
Documentation <i className="fas fa-book" />
</a>
</li>
<li>
<a target="_blank" href="https://slack.min.io">
Ask for help <i className="fa fa-question-circle" />
Ask for help <i className="fas fa-question-circle" />
</a>
</li>
<li>
<a href="" id="show-about" onClick={this.showAbout.bind(this)}>
About <i className="fa fa-info-circle" />
About <i className="fas fa-info-circle" />
</a>
{this.state.showAboutModal && (
<AboutModal
@@ -120,7 +120,7 @@ export class BrowserDropdown extends React.Component {
</li>
<li>
<a href="" onClick={this.showChangePassword.bind(this)}>
Change Password <i className="fa fa-cog" />
Change Password <i className="fas fa-cog" />
</a>
{this.state.showChangePasswordModal && (
<ChangePasswordModal
@@ -131,7 +131,7 @@ export class BrowserDropdown extends React.Component {
</li>
<li>
<a href="" id="logout" onClick={this.logout}>
Sign Out <i className="fa fa-sign-out" />
Sign Out <i className="fas fa-sign-out-alt" />
</a>
</li>
</Dropdown.Menu>
@@ -165,7 +165,7 @@ export class ChangePasswordModal extends React.Component {
})
}}
className={
"toggle-password fa fa-eye " +
"toggle-password fas fa-eye " +
(this.state.currentSecretKeyVisible ? "toggled" : "")
}
/>
@@ -211,7 +211,7 @@ export class ChangePasswordModal extends React.Component {
})
}}
className={
"toggle-password fa fa-eye " +
"toggle-password fas fa-eye " +
(this.state.newSecretKeyVisible ? "toggled" : "")
}
/>
+1 -1
View File
@@ -18,7 +18,7 @@ import React from "react"
export const Host = () => (
<div className="fes-host">
<i className="fa fa-globe" />
<i className="fas fa-globe-americas" />
<a href="/">{window.location.host}</a>
</div>
)
+1 -1
View File
@@ -125,7 +125,7 @@ export class Login extends React.Component {
autoComplete="new-password"
/>
<button className="lw-btn" type="submit">
<i className="fa fa-sign-in" />
<i className="fas fa-sign-in-alt" />
</button>
</form>
</div>
+3 -3
View File
@@ -48,7 +48,7 @@ export const MainActions = ({
<Dropdown dropup className="feb-actions" id="fe-action-toggle">
<Dropdown.Toggle noCaret className="feba-toggle">
<span>
<i className="fa fa-plus" />
<i className="fas fa-plus" />
</span>
</Dropdown.Toggle>
<Dropdown.Menu>
@@ -63,7 +63,7 @@ export const MainActions = ({
/>
<label htmlFor="file-input">
{" "}
<i className="fa fa-cloud-upload" />{" "}
<i className="fas fa-cloud-upload-alt" />{" "}
</label>
</a>
</OverlayTrigger>
@@ -78,7 +78,7 @@ export const MainActions = ({
showMakeBucketModal()
}}
>
<i className="fa fa-hdd-o" />
<i className="far fa-hdd" />
</a>
</OverlayTrigger>
)}
+11 -3
View File
@@ -20,7 +20,6 @@ import ClickOutHandler from "react-onclickout"
import { connect } from "react-redux"
import logo from "../../img/logo.svg"
import Dropdown from "react-bootstrap/lib/Dropdown"
import BucketSearch from "../buckets/BucketSearch"
import BucketList from "../buckets/BucketList"
import Host from "./Host"
@@ -28,8 +27,14 @@ import * as actionsCommon from "./actions"
import web from "../web"
export const SideBar = ({ sidebarOpen, clickOutside }) => {
const onClickOut = e => {
if (e.target.classList.contains("feh-trigger")) {
return
}
clickOutside()
}
return (
<ClickOutHandler onClickOut={clickOutside}>
<ClickOutHandler onClickOut={onClickOut}>
<div
className={classNames({
"fe-sidebar": true,
@@ -62,4 +67,7 @@ const mapDispatchToProps = dispatch => {
}
}
export default connect(mapStateToProps, mapDispatchToProps)(SideBar)
export default connect(
mapStateToProps,
mapDispatchToProps
)(SideBar)
@@ -35,7 +35,20 @@ describe("SideBar", () => {
it("should call clickOutside when the user clicks outside the sidebar", () => {
const clickOutside = jest.fn()
const wrapper = shallow(<SideBar clickOutside={clickOutside} />)
wrapper.simulate("clickOut", { preventDefault: jest.fn() })
wrapper.simulate("clickOut", {
preventDefault: jest.fn(),
target: { classList: { contains: jest.fn(() => false) } }
})
expect(clickOutside).toHaveBeenCalled()
})
it("should not call clickOutside when user clicks on sidebar toggle", () => {
const clickOutside = jest.fn()
const wrapper = shallow(<SideBar clickOutside={clickOutside} />)
wrapper.simulate("clickOut", {
preventDefault: jest.fn(),
target: { classList: { contains: jest.fn(() => true) } }
})
expect(clickOutside).not.toHaveBeenCalled()
})
})
+1 -1
View File
@@ -29,7 +29,7 @@ let BrowserUpdate = ({latestUiVersion}) => {
<a href="">
<OverlayTrigger placement="left" overlay={ <Tooltip id="tt-version-update">
New update available. Click to refresh.
</Tooltip> }> <i className="fa fa-refresh"></i> </OverlayTrigger>
</Tooltip> }> <i className="fas fa-sync"></i> </OverlayTrigger>
</a>
</li>
)
@@ -23,7 +23,7 @@ export const DeleteObjectConfirmModal = ({
}) => (
<ConfirmModal
show={true}
icon="fa fa-exclamation-triangle mci-red"
icon="fas fa-exclamation-triangle mci-red"
text="Are you sure you want to delete?"
sub="This cannot be undone!"
okText="Delete"
+2 -2
View File
@@ -67,14 +67,14 @@ export class ObjectActions extends React.Component {
className="fiad-action"
onClick={this.shareObject.bind(this)}
>
<i className="fa fa-share-alt" />
<i className="fas fa-share-alt" />
</a>
<a
href=""
className="fiad-action"
onClick={this.showDeleteConfirmModal.bind(this)}
>
<i className="fa fa-trash" />
<i className="fas fa-trash-alt" />
</a>
</Dropdown.Menu>
{(showShareObjectModal && shareObjectName === object.name) &&
+2 -2
View File
@@ -59,7 +59,7 @@ export class ObjectsBulkActions extends React.Component {
}
>
<span className="la-label">
<i className="fa fa-check-circle" /> {checkedObjects.length}
<i className="fas fa-check-circle" /> {checkedObjects.length}
{checkedObjects.length === 1 ? " Object " : " Objects "}
selected
</span>
@@ -81,7 +81,7 @@ export class ObjectsBulkActions extends React.Component {
</button>
</span>
<i
className="la-close fa fa-times"
className="la-close fas fa-times"
id="close-bulk-actions"
onClick={clearChecked}
/>
+9 -9
View File
@@ -47,9 +47,9 @@ export const ObjectsHeader = ({
className={classNames({
"fesli-sort": true,
"fesli-sort--active": sortedByName,
fa: true,
"fa-sort-alpha-desc": sortedByName && sortOrder === SORT_ORDER_DESC,
"fa-sort-alpha-asc": sortedByName && sortOrder === SORT_ORDER_ASC
fas: true,
"fa-sort-alpha-down-alt": sortedByName && sortOrder === SORT_ORDER_DESC,
"fa-sort-alpha-down": sortedByName && sortOrder === SORT_ORDER_ASC
})}
/>
</div>
@@ -64,10 +64,10 @@ export const ObjectsHeader = ({
className={classNames({
"fesli-sort": true,
"fesli-sort--active": sortedBySize,
fa: true,
"fa-sort-amount-desc":
fas: true,
"fa-sort-amount-down":
sortedBySize && sortOrder === SORT_ORDER_DESC,
"fa-sort-amount-asc": sortedBySize && sortOrder === SORT_ORDER_ASC
"fa-sort-amount-down-alt": sortedBySize && sortOrder === SORT_ORDER_ASC
})}
/>
</div>
@@ -82,10 +82,10 @@ export const ObjectsHeader = ({
className={classNames({
"fesli-sort": true,
"fesli-sort--active": sortedByLastModified,
fa: true,
"fa-sort-numeric-desc":
fas: true,
"fa-sort-numeric-down-alt":
sortedByLastModified && sortOrder === SORT_ORDER_DESC,
"fa-sort-numeric-asc":
"fa-sort-numeric-down":
sortedByLastModified && sortOrder === SORT_ORDER_ASC
})}
/>
+138 -32
View File
@@ -1,5 +1,5 @@
/*
* MinIO Cloud Storage (C) 2016 MinIO, Inc.
* MinIO Cloud Storage (C) 2016, 2018, 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -16,46 +16,146 @@
import React from "react"
import { connect } from "react-redux"
import ClickOutHandler from "react-onclickout"
import { OverlayTrigger, Tooltip } from "react-bootstrap"
import { getCurrentBucket } from "../buckets/selectors"
import * as actionsObjects from "./actions"
import * as actionsBuckets from "../buckets/actions"
export const Path = ({ currentBucket, currentPrefix, selectPrefix }) => {
const onPrefixClick = (e, prefix) => {
e.preventDefault()
selectPrefix(prefix)
export class Path extends React.Component {
constructor(props) {
super(props)
this.state = {
isEditing: false,
path: ""
}
}
let dirPath = []
let path = ""
if (currentPrefix) {
path = currentPrefix.split("/").map((dir, i) => {
if (dir) {
dirPath.push(dir)
let dirPath_ = dirPath.join("/") + "/"
return (
<span key={i}>
<a href="" onClick={e => onPrefixClick(e, dirPath_)}>
{dir}
</a>
</span>
)
}
stopEditing() {
this.setState({
isEditing: false
})
}
return (
<h2>
<span className="main">
<a onClick={e => onPrefixClick(e, "")} href="">
{currentBucket}
</a>
</span>
{path}
</h2>
)
onPrefixClick(e, prefix) {
e.preventDefault()
const { selectPrefix } = this.props
selectPrefix(prefix)
}
onEditClick(e) {
e.preventDefault()
const { currentBucket, currentPrefix } = this.props
this.setState(
{
isEditing: true,
path: `${currentBucket}/${currentPrefix}`
},
() => {
// focus on input and move cursor to the end
this.pathInput.focus()
this.pathInput.setSelectionRange(
this.state.path.length,
this.state.path.length
)
}
)
}
onKeyDown(e) {
// When Esc key is pressed
if (e.keyCode === 27) {
this.stopEditing()
}
}
onInputClickOut() {
this.stopEditing()
}
bucketExists(bucketName) {
const { buckets } = this.props
return buckets.includes(bucketName)
}
async onSubmit(e) {
e.preventDefault()
const { makeBucket, selectBucket } = this.props
// all paths need to end in slash to display contents properly
let path = this.state.path
if (!path.endsWith("/")) {
path += "/"
}
const splittedPath = path.split("/")
if (splittedPath.length > 0) {
// prevent bucket name from being empty
if (splittedPath[0]) {
const bucketName = splittedPath[0]
const prefix = splittedPath.slice(1).join("/")
if (!this.bucketExists(bucketName)) {
await makeBucket(bucketName)
}
// check updated buckets and don't proceed on invalid inputs
if (this.bucketExists(bucketName)) {
// then select bucket with prefix
selectBucket(bucketName, prefix)
}
this.stopEditing()
}
}
}
render() {
const pathTooltip = <Tooltip id="tt-path">Choose or create new path</Tooltip>
const { currentBucket, currentPrefix } = this.props
let dirPath = []
let path = ""
if (currentPrefix) {
path = currentPrefix.split("/").map((dir, i) => {
if (dir) {
dirPath.push(dir)
let dirPath_ = dirPath.join("/") + "/"
return (
<span key={i}>
<a href="" onClick={e => this.onPrefixClick(e, dirPath_)}>
{dir}
</a>
</span>
)
}
})
}
return (
<h2>
{this.state.isEditing ? (
<ClickOutHandler onClickOut={() => this.onInputClickOut()}>
<form onSubmit={e => this.onSubmit(e)}>
<input
className="form-control form-control--path"
type="text"
placeholder="Choose or create new path"
ref={node => (this.pathInput = node)}
onKeyDown={e => this.onKeyDown(e)}
value={this.state.path}
onChange={e => this.setState({ path: e.target.value })}
/>
</form>
</ClickOutHandler>
) : (
<React.Fragment>
<span className="main">
<a href="" onClick={e => this.onPrefixClick(e, "")}>
{currentBucket}
</a>
</span>
{path}
<OverlayTrigger placement="bottom" overlay={pathTooltip}>
<a href="" onClick={e => this.onEditClick(e)} className="fe-edit">
<i className="fas fa-folder-plus" />
</a>
</OverlayTrigger>
</React.Fragment>
)}
</h2>
)
}
}
const mapStateToProps = state => {
return {
buckets: state.buckets.list,
currentBucket: getCurrentBucket(state),
currentPrefix: state.objects.currentPrefix
}
@@ -63,8 +163,14 @@ const mapStateToProps = state => {
const mapDispatchToProps = dispatch => {
return {
makeBucket: bucket => dispatch(actionsBuckets.makeBucket(bucket)),
selectBucket: (bucket, prefix) =>
dispatch(actionsBuckets.selectBucket(bucket, prefix)),
selectPrefix: prefix => dispatch(actionsObjects.selectPrefix(prefix))
}
}
export default connect(mapStateToProps, mapDispatchToProps)(Path)
export default connect(
mapStateToProps,
mapDispatchToProps
)(Path)
@@ -35,7 +35,7 @@ describe("ObjectsHeader", () => {
/>
)
expect(
wrapper.find("#sort-by-name i").hasClass("fa-sort-alpha-asc")
wrapper.find("#sort-by-name i").hasClass("fa-sort-alpha-down")
).toBeTruthy()
})
@@ -49,7 +49,7 @@ describe("ObjectsHeader", () => {
/>
)
expect(
wrapper.find("#sort-by-name i").hasClass("fa-sort-alpha-desc")
wrapper.find("#sort-by-name i").hasClass("fa-sort-alpha-down-alt")
).toBeTruthy()
})
@@ -63,7 +63,7 @@ describe("ObjectsHeader", () => {
/>
)
expect(
wrapper.find("#sort-by-size i").hasClass("fa-sort-amount-asc")
wrapper.find("#sort-by-size i").hasClass("fa-sort-amount-down-alt")
).toBeTruthy()
})
@@ -77,7 +77,7 @@ describe("ObjectsHeader", () => {
/>
)
expect(
wrapper.find("#sort-by-size i").hasClass("fa-sort-amount-desc")
wrapper.find("#sort-by-size i").hasClass("fa-sort-amount-down")
).toBeTruthy()
})
@@ -91,7 +91,7 @@ describe("ObjectsHeader", () => {
/>
)
expect(
wrapper.find("#sort-by-last-modified i").hasClass("fa-sort-numeric-asc")
wrapper.find("#sort-by-last-modified i").hasClass("fa-sort-numeric-down")
).toBeTruthy()
})
@@ -105,7 +105,7 @@ describe("ObjectsHeader", () => {
/>
)
expect(
wrapper.find("#sort-by-last-modified i").hasClass("fa-sort-numeric-desc")
wrapper.find("#sort-by-last-modified i").hasClass("fa-sort-numeric-down-alt")
).toBeTruthy()
})
+73 -2
View File
@@ -15,7 +15,7 @@
*/
import React from "react"
import { shallow } from "enzyme"
import { shallow, mount } from "enzyme"
import { Path } from "../Path"
describe("Path", () => {
@@ -26,7 +26,12 @@ describe("Path", () => {
it("should render only bucket if there is no prefix", () => {
const wrapper = shallow(<Path currentBucket={"test1"} currentPrefix={""} />)
expect(wrapper.find("span").length).toBe(1)
expect(wrapper.text()).toBe("test1")
expect(
wrapper
.find("span")
.at(0)
.text()
).toBe("test1")
})
it("should render bucket and prefix", () => {
@@ -69,4 +74,70 @@ describe("Path", () => {
.simulate("click", { preventDefault: jest.fn() })
expect(selectPrefix).toHaveBeenCalledWith("a/b/")
})
it("should switch to input mode when edit icon is clicked", () => {
const wrapper = mount(<Path currentBucket={"test1"} currentPrefix={""} />)
wrapper.find(".fe-edit").simulate("click", { preventDefault: jest.fn() })
expect(wrapper.find(".form-control--path").exists()).toBeTruthy()
})
it("should navigate to prefix when user types path for existing bucket", () => {
const selectBucket = jest.fn()
const buckets = ["test1", "test2"]
const wrapper = mount(
<Path
buckets={buckets}
currentBucket={"test1"}
currentPrefix={""}
selectBucket={selectBucket}
/>
)
wrapper.setState({
isEditing: true,
path: "test2/dir1/"
})
wrapper.find("form").simulate("submit", { preventDefault: jest.fn() })
expect(selectBucket).toHaveBeenCalledWith("test2", "dir1/")
})
it("should create a new bucket if bucket typed in path doesn't exist", () => {
const makeBucket = jest.fn()
const buckets = ["test1", "test2"]
const wrapper = mount(
<Path
buckets={buckets}
currentBucket={"test1"}
currentPrefix={""}
makeBucket={makeBucket}
/>
)
wrapper.setState({
isEditing: true,
path: "test3/dir1/"
})
wrapper.find("form").simulate("submit", { preventDefault: jest.fn() })
expect(makeBucket).toHaveBeenCalledWith("test3")
})
it("should not make or select bucket if path doesn't point to bucket", () => {
const makeBucket = jest.fn()
const selectBucket = jest.fn()
const buckets = ["test1", "test2"]
const wrapper = mount(
<Path
buckets={buckets}
currentBucket={"test1"}
currentPrefix={""}
makeBucket={makeBucket}
selectBucket={selectBucket}
/>
)
wrapper.setState({
isEditing: true,
path: "//dir1/dir2/"
})
wrapper.find("form").simulate("submit", { preventDefault: jest.fn() })
expect(makeBucket).not.toHaveBeenCalled()
expect(selectBucket).not.toHaveBeenCalled()
})
})
+4 -4
View File
@@ -33,12 +33,12 @@ export class AbortConfirmModal extends React.Component {
"abort-upload": true
})
let okIcon = classNames({
fa: true,
fas: true,
"fa-times": true
})
let cancelIcon = classNames({
fa: true,
"fa-cloud-upload": true
fas: true,
"fa-cloud-upload-alt": true
})
return (
@@ -46,7 +46,7 @@ export class AbortConfirmModal extends React.Component {
show={true}
baseClass={baseClass}
text="Abort uploads in progress?"
icon="fa fa-info-circle mci-amber"
icon="fas fa-info-circle mci-amber"
sub="This cannot be undone!"
okText="Abort"
okIcon={okIcon}
+9 -1
View File
@@ -8,6 +8,13 @@
background-color: transparent;
}
.form-control--path {
color: @link-color;
padding: 5px 5px 6px 0;
font-size: 16px;
.placeholder(@text-muted-color)
}
select.form-control {
-webkit-appearance: none;
-moz-appearance: none;
@@ -153,7 +160,7 @@ select.form-control {
}
&:after {
content: "\f05a";
font-family: FontAwesome;
font-family: 'Font Awesome 5 Free';
position: absolute;
top: 17px;
right: 9px;
@@ -165,6 +172,7 @@ select.form-control {
.ig-search {
&:before {
font-family: @font-family-icon;
font-weight: 900;
content: '\f002';
font-size: 15px;
position: absolute;
+19 -2
View File
@@ -16,6 +16,10 @@
font-weight: normal;
margin: 0;
@media(min-width: (@screen-md-min)) {
width: calc(100% - 60px);
}
& > span {
margin-bottom: 7px;
display: inline-block;
@@ -27,7 +31,7 @@
color: @text-color;
}
}
&:last-child {
&:last-of-type {
&:after {
content: '/';
margin: 0 4px;
@@ -43,6 +47,19 @@
}
/*--------------------------
Edit path
----------------------------*/
.fe-edit {
font-size: 20px;
color: @link-color;
margin-left: 4px;
i {
vertical-align: middle;
}
}
/*--------------------------
Disk used
----------------------------*/
@@ -132,7 +149,7 @@
@media(max-width: (@screen-sm-max)) {
background: url(../../img/more-h-light.svg) no-repeat center;
.fa-reorder {
.fa-bars {
display: none;
}
}
+5 -4
View File
@@ -113,7 +113,7 @@ div.fesl-row {
Icons
----------------------------*/
&[data-type=folder] {
.list-type(#a1d6dd, '\f114');
.list-type(#a1d6dd, '\f07b');
.fesl-item-name {
a {
@@ -128,8 +128,8 @@ div.fesl-row {
&[data-type=excel] { .list-type(#64c866, '\f1c3'); }
&[data-type=image] { .list-type(#f06292, '\f1c5'); }
&[data-type=video] { .list-type(#f8c363, '\f1c8'); }
&[data-type=other] { .list-type(#afafaf, '\f016'); }
&[data-type=text] { .list-type(#8a8a8a, '\f0f6'); }
&[data-type=other] { .list-type(#afafaf, '\f15b'); }
&[data-type=text] { .list-type(#8a8a8a, '\f15c'); }
&[data-type=doc] { .list-type(#2196f5, '\f1c2'); }
&[data-type=presentation] { .list-type(#896ea6, '\f1c4'); }
@@ -249,6 +249,7 @@ div.fesl-row {
&:after {
font-family: @font-family-icon;
font-weight: 900;
content: '\f00c';
top: 8px;
left: 9px;
@@ -449,7 +450,7 @@ div.fesl-row {
float: left;
padding: 4px 0;
.fa {
.fas {
font-size: 22px;
vertical-align: top;
margin-right: 10px;
+1 -1
View File
@@ -76,7 +76,7 @@
word-wrap: break-word;
&:before {
font-family: FontAwesome;
font-family: 'Font Awesome 5 Free';
content: '\f0a0';
font-size: 17px;
position: absolute;
+1 -1
View File
@@ -2,7 +2,7 @@
Base
----------------------------*/
@font-family-sans-serif : 'Lato', sans-serif;
@font-family-icon : 'fontAwesome';
@font-family-icon : 'Font Awesome 5 Free';
@body-bg : #edecec;
@text-color : #8e8e8e;
@font-size-base : 15px;
+8 -8
View File
@@ -82,6 +82,11 @@
}
}
},
"@fortawesome/fontawesome-free": {
"version": "5.10.0",
"resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-free/-/fontawesome-free-5.10.0.tgz",
"integrity": "sha512-XX16koDMY/tkJmec0VFfKF7RYZOze/203B1iyLnRaAySm3ZPhKaeyIpf73Yh8xhrMk3Fj4TeH3FC01qyFTyg8g=="
},
"@types/node": {
"version": "12.0.7",
"resolved": "https://registry.npmjs.org/@types/node/-/node-12.0.7.tgz",
@@ -4571,11 +4576,6 @@
}
}
},
"font-awesome": {
"version": "4.7.0",
"resolved": "https://registry.npmjs.org/font-awesome/-/font-awesome-4.7.0.tgz",
"integrity": "sha1-j6jPBBGhoxr9B7BtKQK7n8gVoTM="
},
"for-in": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/for-in/-/for-in-1.0.2.tgz",
@@ -10403,9 +10403,9 @@
"integrity": "sha512-fBASbA6LnOU9dOU2eW7aQ8xmYBSXUIWr+UmF9b1efZBazGNO+rcXT/icdKnYm2pTwcRylVUYwW7H1PHfLekVzA=="
},
"react-onclickout": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/react-onclickout/-/react-onclickout-2.0.4.tgz",
"integrity": "sha1-LHU5pkfh3NyrCyji9Orjw+APDGQ="
"version": "2.0.8",
"resolved": "https://registry.npmjs.org/react-onclickout/-/react-onclickout-2.0.8.tgz",
"integrity": "sha1-0XixP7h6SBNWdhtFSqYN9wabLaQ="
},
"react-overlays": {
"version": "0.8.3",
+2 -2
View File
@@ -59,10 +59,10 @@
"webpack-dev-server": "^3.1.14"
},
"dependencies": {
"@fortawesome/fontawesome-free": "^5.10.0",
"bootstrap": "^3.3.6",
"classnames": "^2.2.3",
"expect": "^1.20.2",
"font-awesome": "^4.7.0",
"glob-all": "^3.1.0",
"history": "^4.7.2",
"humanize": "0.0.9",
@@ -82,7 +82,7 @@
"react-dom": "^16.2.0",
"react-dropzone": "^4.2.3",
"react-infinite-scroller": "^1.0.6",
"react-onclickout": "2.0.4",
"react-onclickout": "^2.0.8",
"react-redux": "^5.0.6",
"react-router-dom": "^4.2.0",
"redux": "^3.7.2",
+40 -32
View File
File diff suppressed because one or more lines are too long
+5 -4
View File
@@ -10,6 +10,11 @@
esutils "^2.0.2"
js-tokens "^3.0.0"
"@fortawesome/fontawesome-free@^5.10.0":
version "5.10.0"
resolved "https://registry.yarnpkg.com/@fortawesome/fontawesome-free/-/fontawesome-free-5.10.0.tgz#dbfb34093c87c9516f03ee1db3385adae3ae9461"
integrity sha512-XX16koDMY/tkJmec0VFfKF7RYZOze/203B1iyLnRaAySm3ZPhKaeyIpf73Yh8xhrMk3Fj4TeH3FC01qyFTyg8g==
"@types/node@*":
version "9.4.0"
resolved "https://registry.yarnpkg.com/@types/node/-/node-9.4.0.tgz#b85a0bcf1e1cc84eb4901b7e96966aedc6f078d1"
@@ -3230,10 +3235,6 @@ follow-redirects@^1.0.0:
dependencies:
debug "=3.1.0"
font-awesome@^4.7.0:
version "4.7.0"
resolved "https://registry.yarnpkg.com/font-awesome/-/font-awesome-4.7.0.tgz#8fa8cf0411a1a31afd07b06d2902bb9fc815a133"
for-in@^1.0.1, for-in@^1.0.2:
version "1.0.2"
resolved "https://registry.yarnpkg.com/for-in/-/for-in-1.0.2.tgz#81068d295a8142ec0ac726c6e2200c30fb6d5e80"
+1
View File
@@ -23,6 +23,7 @@ function _build() {
export GOOS=$os
export GOARCH=$arch
export GO111MODULE=on
export GOPROXY=https://proxy.golang.org
go build -tags kqueue -o /dev/null
}
+1 -1
View File
@@ -2,4 +2,4 @@
set -e
GO111MODULE=on CGO_ENABLED=0 go test -v -coverprofile=coverage.txt -covermode=atomic ./...
GOPROXY=https://proxy.golang.org GO111MODULE=on CGO_ENABLED=0 go test -v -coverprofile=coverage.txt -covermode=atomic ./...
+1
View File
@@ -33,6 +33,7 @@ export ACCESS_KEY="minio"
export SECRET_KEY="minio123"
export ENABLE_HTTPS=0
export GO111MODULE=on
export GOPROXY=https://proxy.golang.org
MINIO_CONFIG_DIR="$WORK_DIR/.minio"
MINIO=( "$PWD/minio" --config-dir "$MINIO_CONFIG_DIR" )
+192 -34
View File
@@ -23,6 +23,7 @@ import (
"encoding/json"
"errors"
"io"
"io/ioutil"
"net/http"
"os"
"sort"
@@ -1019,6 +1020,157 @@ func (a adminAPIHandlers) ListUsers(w http.ResponseWriter, r *http.Request) {
writeSuccessResponseJSON(w, econfigData)
}
// GetUserInfo - GET /minio/admin/v1/user-info
func (a adminAPIHandlers) GetUserInfo(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "GetUserInfo")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
return
}
vars := mux.Vars(r)
name := vars["accessKey"]
userInfo, err := globalIAMSys.GetUserInfo(name)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
data, err := json.Marshal(userInfo)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
writeSuccessResponseJSON(w, data)
}
// UpdateGroupMembers - PUT /minio/admin/v1/update-group-members
func (a adminAPIHandlers) UpdateGroupMembers(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "UpdateGroupMembers")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
return
}
defer r.Body.Close()
data, err := ioutil.ReadAll(r.Body)
if err != nil {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrInvalidRequest), r.URL)
return
}
var updReq madmin.GroupAddRemove
err = json.Unmarshal(data, &updReq)
if err != nil {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrInvalidRequest), r.URL)
return
}
if updReq.IsRemove {
err = globalIAMSys.RemoveUsersFromGroup(updReq.Group, updReq.Members)
} else {
err = globalIAMSys.AddUsersToGroup(updReq.Group, updReq.Members)
}
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
// Notify all other MinIO peers to load group.
for _, nerr := range globalNotificationSys.LoadGroup(updReq.Group) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
}
}
}
// GetGroup - /minio/admin/v1/group?group=mygroup1
func (a adminAPIHandlers) GetGroup(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "GetGroup")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
return
}
vars := mux.Vars(r)
group := vars["group"]
gdesc, err := globalIAMSys.GetGroupDescription(group)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
body, err := json.Marshal(gdesc)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
writeSuccessResponseJSON(w, body)
}
// ListGroups - GET /minio/admin/v1/groups
func (a adminAPIHandlers) ListGroups(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "ListGroups")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
return
}
groups := globalIAMSys.ListGroups()
body, err := json.Marshal(groups)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
writeSuccessResponseJSON(w, body)
}
// SetGroupStatus - PUT /minio/admin/v1/set-group-status?group=mygroup1&status=enabled
func (a adminAPIHandlers) SetGroupStatus(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "SetGroupStatus")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
return
}
vars := mux.Vars(r)
group := vars["group"]
status := vars["status"]
var err error
if status == statusEnabled {
err = globalIAMSys.SetGroupStatus(group, true)
} else if status == statusDisabled {
err = globalIAMSys.SetGroupStatus(group, false)
} else {
err = errInvalidArgument
}
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
// Notify all other MinIO peers to reload user.
for _, nerr := range globalNotificationSys.LoadGroup(group) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
}
}
}
// SetUserStatus - PUT /minio/admin/v1/set-user-status?accessKey=<access_key>&status=[enabled|disabled]
func (a adminAPIHandlers) SetUserStatus(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "SetUserStatus")
@@ -1228,9 +1380,9 @@ func (a adminAPIHandlers) AddCannedPolicy(w http.ResponseWriter, r *http.Request
}
}
// SetUserPolicy - PUT /minio/admin/v1/set-user-policy?accessKey=<access_key>&name=<policy_name>
func (a adminAPIHandlers) SetUserPolicy(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "SetUserPolicy")
// SetPolicyForUserOrGroup - PUT /minio/admin/v1/set-policy?policy=xxx&user-or-group=?[&is-group]
func (a adminAPIHandlers) SetPolicyForUserOrGroup(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "SetPolicyForUserOrGroup")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
@@ -1238,8 +1390,9 @@ func (a adminAPIHandlers) SetUserPolicy(w http.ResponseWriter, r *http.Request)
}
vars := mux.Vars(r)
accessKey := vars["accessKey"]
policyName := vars["name"]
policyName := vars["policyName"]
entityName := vars["userOrGroup"]
isGroup := vars["isGroup"] == "true"
// Deny if WORM is enabled
if globalWORMEnabled {
@@ -1247,18 +1400,13 @@ func (a adminAPIHandlers) SetUserPolicy(w http.ResponseWriter, r *http.Request)
return
}
// Custom IAM policies not allowed for admin user.
if accessKey == globalServerConfig.GetCredential().AccessKey {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrInvalidRequest), r.URL)
if err := globalIAMSys.PolicyDBSet(entityName, policyName, isGroup); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
if err := globalIAMSys.SetUserPolicy(accessKey, policyName); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
}
// Notify all other Minio peers to reload user
for _, nerr := range globalNotificationSys.LoadUser(accessKey, false) {
// Notify all other MinIO peers to reload policy
for _, nerr := range globalNotificationSys.LoadPolicyMapping(entityName, isGroup) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
@@ -1459,6 +1607,23 @@ func (a adminAPIHandlers) SetConfigKeysHandler(w http.ResponseWriter, r *http.Re
writeSuccessResponseHeadersOnly(w)
}
// Returns true if the trace.Info should be traced,
// false if certain conditions are not met.
// - input entry is not of the type *trace.Info*
// - errOnly entries are to be traced, not status code 2xx, 3xx.
// - all entries to be traced, if not trace only S3 API requests.
func mustTrace(entry interface{}, trcAll, errOnly bool) bool {
trcInfo, ok := entry.(trace.Info)
if !ok {
return false
}
trace := trcAll || !hasPrefix(trcInfo.ReqInfo.Path, minioReservedBucketPath+SlashSeparator)
if errOnly {
return trace && trcInfo.RespInfo.StatusCode >= http.StatusBadRequest
}
return trace
}
// TraceHandler - POST /minio/admin/v1/trace
// ----------
// The handler sends http trace to the connected HTTP client.
@@ -1474,10 +1639,6 @@ func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
return
}
// Avoid reusing tcp connection if read timeout is hit
// This is needed to make r.Context().Done() work as
// expected in case of read timeout
w.Header().Set(xhttp.Connection, "close")
w.Header().Set(xhttp.ContentType, "text/event-stream")
doneCh := make(chan struct{})
@@ -1487,28 +1648,22 @@ func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
// Use buffered channel to take care of burst sends or slow w.Write()
traceCh := make(chan interface{}, 4000)
filter := func(entry interface{}) bool {
trcInfo := entry.(trace.Info)
if trcErr && isHTTPStatusOK(trcInfo.RespInfo.StatusCode) {
return false
}
if trcAll {
return true
}
return !strings.HasPrefix(trcInfo.ReqInfo.Path, minioReservedBucketPath)
}
remoteHosts := getRemoteHosts(globalEndpoints)
peers, err := getRestClients(remoteHosts)
peers, err := getRestClients(getRemoteHosts(globalEndpoints))
if err != nil {
return
}
globalHTTPTrace.Subscribe(traceCh, doneCh, filter)
globalHTTPTrace.Subscribe(traceCh, doneCh, func(entry interface{}) bool {
return mustTrace(entry, trcAll, trcErr)
})
for _, peer := range peers {
peer.Trace(traceCh, doneCh, trcAll, trcErr)
}
keepAliveTicker := time.NewTicker(500 * time.Millisecond)
defer keepAliveTicker.Stop()
enc := json.NewEncoder(w)
for {
select {
@@ -1517,8 +1672,11 @@ func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
return
}
w.(http.Flusher).Flush()
case <-r.Context().Done():
return
case <-keepAliveTicker.C:
if _, err := w.Write([]byte(" ")); err != nil {
return
}
w.(http.Flusher).Flush()
case <-GlobalServiceDoneCh:
return
}
+9 -3
View File
@@ -83,7 +83,9 @@ var (
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
"autoDeleted": false,
"queueDir": "",
"queueLimit": 0
}
},
"elasticsearch": {
@@ -139,7 +141,9 @@ var (
"port": "",
"user": "",
"password": "",
"database": ""
"database": "",
"queueDir": "",
"queueLimit": 0
}
},
"nats": {
@@ -185,7 +189,9 @@ var (
"port": "",
"user": "",
"password": "",
"database": ""
"database": "",
"queueDir": "",
"queueLimit": 0
}
},
"redis": {
+3 -3
View File
@@ -587,9 +587,9 @@ func (h *healSequence) healItemsFromSourceCh() error {
var itemType madmin.HealItemType
switch {
case path == "/":
case path == SlashSeparator:
itemType = madmin.HealItemMetadata
case !strings.Contains(path, "/"):
case !strings.Contains(path, SlashSeparator):
itemType = madmin.HealItemBucket
default:
itemType = madmin.HealItemObject
@@ -693,7 +693,7 @@ func (h *healSequence) healDiskFormat() error {
return errServerNotInitialized
}
return h.queueHealTask("/", madmin.HealItemMetadata)
return h.queueHealTask(SlashSeparator, madmin.HealItemMetadata)
}
// healBuckets - check for all buckets heal or just particular bucket.
+20 -2
View File
@@ -96,20 +96,38 @@ func registerAdminRouter(router *mux.Router, enableConfigOps, enableIAMOps bool)
// Add user IAM
adminV1Router.Methods(http.MethodPut).Path("/add-user").HandlerFunc(httpTraceHdrs(adminAPI.AddUser)).Queries("accessKey", "{accessKey:.*}")
adminV1Router.Methods(http.MethodPut).Path("/set-user-policy").HandlerFunc(httpTraceHdrs(adminAPI.SetUserPolicy)).
Queries("accessKey", "{accessKey:.*}").Queries("name", "{name:.*}")
adminV1Router.Methods(http.MethodPut).Path("/set-user-status").HandlerFunc(httpTraceHdrs(adminAPI.SetUserStatus)).
Queries("accessKey", "{accessKey:.*}").Queries("status", "{status:.*}")
// Remove policy IAM
adminV1Router.Methods(http.MethodDelete).Path("/remove-canned-policy").HandlerFunc(httpTraceHdrs(adminAPI.RemoveCannedPolicy)).Queries("name", "{name:.*}")
// Set user or group policy
adminV1Router.Methods(http.MethodPut).Path("/set-user-or-group-policy").
HandlerFunc(httpTraceHdrs(adminAPI.SetPolicyForUserOrGroup)).
Queries("policyName", "{policyName:.*}", "userOrGroup", "{userOrGroup:.*}", "isGroup", "{isGroup:true|false}")
// Remove user IAM
adminV1Router.Methods(http.MethodDelete).Path("/remove-user").HandlerFunc(httpTraceHdrs(adminAPI.RemoveUser)).Queries("accessKey", "{accessKey:.*}")
// List users
adminV1Router.Methods(http.MethodGet).Path("/list-users").HandlerFunc(httpTraceHdrs(adminAPI.ListUsers))
// User info
adminV1Router.Methods(http.MethodGet).Path("/user-info").HandlerFunc(httpTraceHdrs(adminAPI.GetUserInfo)).Queries("accessKey", "{accessKey:.*}")
// Add/Remove members from group
adminV1Router.Methods(http.MethodPut).Path("/update-group-members").HandlerFunc(httpTraceHdrs(adminAPI.UpdateGroupMembers))
// Get Group
adminV1Router.Methods(http.MethodGet).Path("/group").HandlerFunc(httpTraceHdrs(adminAPI.GetGroup)).Queries("group", "{group:.*}")
// List Groups
adminV1Router.Methods(http.MethodGet).Path("/groups").HandlerFunc(httpTraceHdrs(adminAPI.ListGroups))
// Set Group Status
adminV1Router.Methods(http.MethodPut).Path("/set-group-status").HandlerFunc(httpTraceHdrs(adminAPI.SetGroupStatus)).Queries("group", "{group:.*}").Queries("status", "{status:.*}")
// List policies
adminV1Router.Methods(http.MethodGet).Path("/list-canned-policies").HandlerFunc(httpTraceHdrs(adminAPI.ListCannedPolicies))
}
+16
View File
@@ -203,6 +203,8 @@ const (
ErrMalformedJSON
ErrAdminNoSuchUser
ErrAdminNoSuchGroup
ErrAdminGroupNotEmpty
ErrAdminNoSuchPolicy
ErrAdminInvalidArgument
ErrAdminInvalidAccessKey
@@ -923,6 +925,16 @@ var errorCodes = errorCodeMap{
Description: "The specified user does not exist.",
HTTPStatusCode: http.StatusNotFound,
},
ErrAdminNoSuchGroup: {
Code: "XMinioAdminNoSuchGroup",
Description: "The specified group does not exist.",
HTTPStatusCode: http.StatusNotFound,
},
ErrAdminGroupNotEmpty: {
Code: "XMinioAdminGroupNotEmpty",
Description: "The specified group is not empty - cannot remove it.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrAdminNoSuchPolicy: {
Code: "XMinioAdminNoSuchPolicy",
Description: "The canned policy does not exist.",
@@ -1500,6 +1512,10 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrAdminInvalidArgument
case errNoSuchUser:
apiErr = ErrAdminNoSuchUser
case errNoSuchGroup:
apiErr = ErrAdminNoSuchGroup
case errGroupNotEmpty:
apiErr = ErrAdminGroupNotEmpty
case errNoSuchPolicy:
apiErr = ErrAdminNoSuchPolicy
case errSignatureMismatch:
+9 -9
View File
@@ -36,7 +36,7 @@ func TestListObjectsV2Resources(t *testing.T) {
"prefix": []string{"photos/"},
"continuation-token": []string{"token"},
"start-after": []string{"start-after"},
"delimiter": []string{"/"},
"delimiter": []string{SlashSeparator},
"fetch-owner": []string{"true"},
"max-keys": []string{"100"},
"encoding-type": []string{"gzip"},
@@ -44,7 +44,7 @@ func TestListObjectsV2Resources(t *testing.T) {
prefix: "photos/",
token: "token",
startAfter: "start-after",
delimiter: "/",
delimiter: SlashSeparator,
fetchOwner: true,
maxKeys: 100,
encodingType: "gzip",
@@ -55,14 +55,14 @@ func TestListObjectsV2Resources(t *testing.T) {
"prefix": []string{"photos/"},
"continuation-token": []string{"token"},
"start-after": []string{"start-after"},
"delimiter": []string{"/"},
"delimiter": []string{SlashSeparator},
"fetch-owner": []string{"true"},
"encoding-type": []string{"gzip"},
},
prefix: "photos/",
token: "token",
startAfter: "start-after",
delimiter: "/",
delimiter: SlashSeparator,
fetchOwner: true,
maxKeys: 1000,
encodingType: "gzip",
@@ -73,7 +73,7 @@ func TestListObjectsV2Resources(t *testing.T) {
"prefix": []string{"photos/"},
"continuation-token": []string{""},
"start-after": []string{"start-after"},
"delimiter": []string{"/"},
"delimiter": []string{SlashSeparator},
"fetch-owner": []string{"true"},
"encoding-type": []string{"gzip"},
},
@@ -130,13 +130,13 @@ func TestListObjectsV1Resources(t *testing.T) {
values: url.Values{
"prefix": []string{"photos/"},
"marker": []string{"test"},
"delimiter": []string{"/"},
"delimiter": []string{SlashSeparator},
"max-keys": []string{"100"},
"encoding-type": []string{"gzip"},
},
prefix: "photos/",
marker: "test",
delimiter: "/",
delimiter: SlashSeparator,
maxKeys: 100,
encodingType: "gzip",
},
@@ -144,12 +144,12 @@ func TestListObjectsV1Resources(t *testing.T) {
values: url.Values{
"prefix": []string{"photos/"},
"marker": []string{"test"},
"delimiter": []string{"/"},
"delimiter": []string{SlashSeparator},
"encoding-type": []string{"gzip"},
},
prefix: "photos/",
marker: "test",
delimiter: "/",
delimiter: SlashSeparator,
maxKeys: 1000,
encodingType: "gzip",
},
+2 -2
View File
@@ -293,14 +293,14 @@ func getObjectLocation(r *http.Request, domains []string, bucket, object string)
}
u := &url.URL{
Host: r.Host,
Path: path.Join(slashSeparator, bucket, object),
Path: path.Join(SlashSeparator, bucket, object),
Scheme: proto,
}
// If domain is set then we need to use bucket DNS style.
for _, domain := range domains {
if strings.Contains(r.Host, domain) {
u.Host = bucket + "." + r.Host
u.Path = path.Join(slashSeparator, object)
u.Path = path.Join(SlashSeparator, object)
break
}
}
+2 -2
View File
@@ -48,7 +48,7 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled, allowSSEKMS bool)
}
// API Router
apiRouter := router.PathPrefix("/").Subrouter()
apiRouter := router.PathPrefix(SlashSeparator).Subrouter()
var routers []*mux.Router
for _, domainName := range globalDomainNames {
routers = append(routers, apiRouter.Host("{bucket:.+}."+domainName).Subrouter())
@@ -157,7 +157,7 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled, allowSSEKMS bool)
/// Root operation
// ListBuckets
apiRouter.Methods(http.MethodGet).Path("/").HandlerFunc(httpTraceAll(api.ListBucketsHandler))
apiRouter.Methods(http.MethodGet).Path(SlashSeparator).HandlerFunc(httpTraceAll(api.ListBucketsHandler))
// If none of the routes match.
apiRouter.NotFoundHandler = http.HandlerFunc(httpTraceAll(notFoundHandler))
+26 -13
View File
@@ -259,14 +259,24 @@ func checkClaimsFromToken(r *http.Request, cred auth.Credentials) (map[string]in
// for authenticated requests validates IAM policies.
// returns APIErrorCode if any to be replied to the client.
func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Action, bucketName, objectName string) (s3Err APIErrorCode) {
_, _, s3Err = checkRequestAuthTypeToAccessKey(ctx, r, action, bucketName, objectName)
return s3Err
}
// Check request auth type verifies the incoming http request
// - validates the request signature
// - validates the policy action if anonymous tests bucket policies if any,
// for authenticated requests validates IAM policies.
// returns APIErrorCode if any to be replied to the client.
// Additionally returns the accessKey used in the request, and if this request is by an admin.
func checkRequestAuthTypeToAccessKey(ctx context.Context, r *http.Request, action policy.Action, bucketName, objectName string) (accessKey string, owner bool, s3Err APIErrorCode) {
var cred auth.Credentials
var owner bool
switch getRequestAuthType(r) {
case authTypeUnknown, authTypeStreamingSigned:
return ErrAccessDenied
return accessKey, owner, ErrAccessDenied
case authTypePresignedV2, authTypeSignedV2:
if s3Err = isReqAuthenticatedV2(r); s3Err != ErrNone {
return s3Err
return accessKey, owner, s3Err
}
cred, owner, s3Err = getReqAccessKeyV2(r)
case authTypeSigned, authTypePresigned:
@@ -276,17 +286,18 @@ func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Ac
region = ""
}
if s3Err = isReqAuthenticated(ctx, r, region, serviceS3); s3Err != ErrNone {
return s3Err
return accessKey, owner, s3Err
}
cred, owner, s3Err = getReqAccessKeyV4(r, region, serviceS3)
}
if s3Err != ErrNone {
return s3Err
return accessKey, owner, s3Err
}
claims, s3Err := checkClaimsFromToken(r, cred)
var claims map[string]interface{}
claims, s3Err = checkClaimsFromToken(r, cred)
if s3Err != ErrNone {
return s3Err
return accessKey, owner, s3Err
}
// LocationConstraint is valid only for CreateBucketAction.
@@ -296,7 +307,7 @@ func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Ac
payload, err := ioutil.ReadAll(io.LimitReader(r.Body, maxLocationConstraintSize))
if err != nil {
logger.LogIf(ctx, err)
return ErrMalformedXML
return accessKey, owner, ErrMalformedXML
}
// Populate payload to extract location constraint.
@@ -305,7 +316,7 @@ func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Ac
var s3Error APIErrorCode
locationConstraint, s3Error = parseLocationConstraint(r)
if s3Error != ErrNone {
return s3Error
return accessKey, owner, s3Error
}
// Populate payload again to handle it in HTTP handler.
@@ -321,9 +332,10 @@ func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Ac
IsOwner: false,
ObjectName: objectName,
}) {
return ErrNone
// Request is allowed return the appropriate access key.
return cred.AccessKey, owner, ErrNone
}
return ErrAccessDenied
return accessKey, owner, ErrAccessDenied
}
if globalIAMSys.IsAllowed(iampolicy.Args{
@@ -335,9 +347,10 @@ func checkRequestAuthType(ctx context.Context, r *http.Request, action policy.Ac
IsOwner: owner,
Claims: claims,
}) {
return ErrNone
// Request is allowed return the appropriate access key.
return cred.AccessKey, owner, ErrNone
}
return ErrAccessDenied
return accessKey, owner, ErrAccessDenied
}
// Verify if request has valid AWS Signature Version '2'.
+5 -5
View File
@@ -44,7 +44,7 @@ func TestGetRequestAuthType(t *testing.T) {
URL: &url.URL{
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
Path: SlashSeparator,
},
Header: http.Header{
"Authorization": []string{"AWS4-HMAC-SHA256 <cred_string>"},
@@ -62,7 +62,7 @@ func TestGetRequestAuthType(t *testing.T) {
URL: &url.URL{
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
Path: SlashSeparator,
},
Header: http.Header{
"Authorization": []string{"Bearer 12313123"},
@@ -77,7 +77,7 @@ func TestGetRequestAuthType(t *testing.T) {
URL: &url.URL{
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
Path: SlashSeparator,
},
Header: http.Header{
"Authorization": []string{""},
@@ -92,7 +92,7 @@ func TestGetRequestAuthType(t *testing.T) {
URL: &url.URL{
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
Path: SlashSeparator,
RawQuery: "X-Amz-Credential=EXAMPLEINVALIDEXAMPL%2Fs3%2F20160314%2Fus-east-1",
},
},
@@ -105,7 +105,7 @@ func TestGetRequestAuthType(t *testing.T) {
URL: &url.URL{
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
Path: SlashSeparator,
},
Header: http.Header{
"Content-Type": []string{"multipart/form-data"},
+1 -6
View File
@@ -93,9 +93,7 @@ func (api objectAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *http
}
listObjectsV2 := objectAPI.ListObjectsV2
if api.CacheAPI() != nil {
listObjectsV2 = api.CacheAPI().ListObjectsV2
}
// Inititate a list objects operation based on the input params.
// On success would return back ListObjectsInfo object to be
// marshaled into S3 compatible XML header.
@@ -172,9 +170,6 @@ func (api objectAPIHandlers) ListObjectsV1Handler(w http.ResponseWriter, r *http
}
listObjects := objectAPI.ListObjects
if api.CacheAPI() != nil {
listObjects = api.CacheAPI().ListObjects
}
// Inititate a list objects operation based on the input params.
// On success would return back ListObjectsInfo object to be
+73 -16
View File
@@ -22,6 +22,7 @@ import (
"encoding/xml"
"fmt"
"io"
"net"
"net/http"
"net/url"
"path"
@@ -38,6 +39,7 @@ import (
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/handlers"
"github.com/minio/minio/pkg/hash"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/policy"
"github.com/minio/minio/pkg/sync/errgroup"
)
@@ -50,15 +52,27 @@ import (
// -- If yes, check if the IP of entry matches local IP. This means entry is for this instance.
// -- If IP of the entry doesn't match, this means entry is for another instance. Log an error to console.
func initFederatorBackend(objLayer ObjectLayer) {
// Get buckets in the backend
b, err := objLayer.ListBuckets(context.Background())
if err != nil {
logger.LogIf(context.Background(), err)
return
}
// Get buckets in the DNS
dnsBuckets, err := globalDNSConfig.List()
if err != nil && err != dns.ErrNoEntriesFound {
logger.LogIf(context.Background(), err)
return
}
bucketSet := set.NewStringSet()
// Add buckets that are not registered with the DNS
g := errgroup.WithNErrs(len(b))
for index := range b {
index := index
bucketSet.Add(b[index].Name)
g.Go(func() error {
r, gerr := globalDNSConfig.Get(b[index].Name)
if gerr != nil {
@@ -78,7 +92,38 @@ func initFederatorBackend(objLayer ObjectLayer) {
for _, err := range g.Wait() {
if err != nil {
logger.LogIf(context.Background(), err)
return
}
}
g = errgroup.WithNErrs(len(dnsBuckets))
// Remove buckets that are in DNS for this server, but aren't local
for index := range dnsBuckets {
index := index
g.Go(func() error {
// This is a local bucket that exists, so we can continue
if bucketSet.Contains(dnsBuckets[index].Key) {
return nil
}
// This is not for our server, so we can continue
hostPort := net.JoinHostPort(dnsBuckets[index].Host, fmt.Sprintf("%d", dnsBuckets[index].Port))
if globalDomainIPs.Intersection(set.CreateStringSet(hostPort)).IsEmpty() {
return nil
}
// We go to here, so we know the bucket no longer exists, but is registered in DNS to this server
if err := globalDNSConfig.DeleteRecord(dnsBuckets[index]); err != nil {
return fmt.Errorf("Failed to remove DNS entry for %s due to %v", dnsBuckets[index].Key, err)
}
return nil
}, index)
}
for _, err := range g.Wait() {
if err != nil {
logger.LogIf(context.Background(), err)
}
}
}
@@ -106,9 +151,7 @@ func (api objectAPIHandlers) GetBucketLocationHandler(w http.ResponseWriter, r *
}
getBucketInfo := objectAPI.GetBucketInfo
if api.CacheAPI() != nil {
getBucketInfo = api.CacheAPI().GetBucketInfo
}
if _, err := getBucketInfo(ctx, bucket); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
@@ -203,11 +246,9 @@ func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.R
}
listBuckets := objectAPI.ListBuckets
if api.CacheAPI() != nil {
listBuckets = api.CacheAPI().ListBuckets
}
if s3Error := checkRequestAuthType(ctx, r, policy.ListAllMyBucketsAction, "", ""); s3Error != ErrNone {
accessKey, owner, s3Error := checkRequestAuthTypeToAccessKey(ctx, r, policy.ListAllMyBucketsAction, "", "")
if s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
@@ -241,8 +282,28 @@ func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.R
}
}
// Set prefix value for "s3:prefix" policy conditionals.
r.Header.Set("prefix", "")
// Set delimiter value for "s3:delimiter" policy conditionals.
r.Header.Set("delimiter", SlashSeparator)
var newBucketsInfo []BucketInfo
for _, bucketInfo := range bucketsInfo {
if globalIAMSys.IsAllowed(iampolicy.Args{
AccountName: accessKey,
Action: iampolicy.ListBucketAction,
BucketName: bucketInfo.Name,
ConditionValues: getConditionValues(r, "", accessKey),
IsOwner: owner,
ObjectName: "",
}) {
newBucketsInfo = append(newBucketsInfo, bucketInfo)
}
}
// Generate response.
response := generateListBucketsResponse(bucketsInfo)
response := generateListBucketsResponse(newBucketsInfo)
encodedSuccessResponse := encodeResponse(response)
// Write response.
@@ -650,7 +711,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
if objectAPI.IsEncryptionSupported() {
if hasServerSideEncryptionHeader(formValues) && !hasSuffix(object, slashSeparator) { // handle SSE-C and SSE-S3 requests
if hasServerSideEncryptionHeader(formValues) && !hasSuffix(object, SlashSeparator) { // handle SSE-C and SSE-S3 requests
var reader io.Reader
var key []byte
if crypto.SSEC.IsRequested(formValues) {
@@ -747,9 +808,7 @@ func (api objectAPIHandlers) HeadBucketHandler(w http.ResponseWriter, r *http.Re
}
getBucketInfo := objectAPI.GetBucketInfo
if api.CacheAPI() != nil {
getBucketInfo = api.CacheAPI().GetBucketInfo
}
if _, err := getBucketInfo(ctx, bucket); err != nil {
writeErrorResponseHeadersOnly(w, toAPIError(ctx, err))
return
@@ -779,9 +838,7 @@ func (api objectAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http.
}
deleteBucket := objectAPI.DeleteBucket
if api.CacheAPI() != nil {
deleteBucket = api.CacheAPI().DeleteBucket
}
// Attempt to delete bucket.
if err := deleteBucket(ctx, bucket); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
+2 -2
View File
@@ -69,7 +69,7 @@ func testGetBucketLocationHandler(obj ObjectLayer, instanceType, bucketName stri
expectedRespStatus: http.StatusForbidden,
locationResponse: []byte(""),
errorResponse: APIErrorResponse{
Resource: "/" + bucketName + "/",
Resource: SlashSeparator + bucketName + SlashSeparator,
Code: "InvalidAccessKeyId",
Message: "The access key ID you provided does not exist in our records.",
},
@@ -394,7 +394,7 @@ func testListMultipartUploadsHandler(obj ObjectLayer, instanceType, bucketName s
prefix: "",
keyMarker: "",
uploadIDMarker: "",
delimiter: "/",
delimiter: SlashSeparator,
maxUploads: "100",
accessKey: credentials.AccessKey,
secretKey: credentials.SecretKey,
+4 -3
View File
@@ -24,6 +24,7 @@ import (
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lifecycle"
"github.com/minio/minio/pkg/policy"
)
const (
@@ -47,7 +48,7 @@ func (api objectAPIHandlers) PutBucketLifecycleHandler(w http.ResponseWriter, r
vars := mux.Vars(r)
bucket := vars["bucket"]
if s3Error := checkRequestAuthType(ctx, r, lifecycle.PutBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, policy.PutBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
@@ -97,7 +98,7 @@ func (api objectAPIHandlers) GetBucketLifecycleHandler(w http.ResponseWriter, r
vars := mux.Vars(r)
bucket := vars["bucket"]
if s3Error := checkRequestAuthType(ctx, r, lifecycle.GetBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, policy.GetBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
@@ -140,7 +141,7 @@ func (api objectAPIHandlers) DeleteBucketLifecycleHandler(w http.ResponseWriter,
vars := mux.Vars(r)
bucket := vars["bucket"]
if s3Error := checkRequestAuthType(ctx, r, lifecycle.DeleteBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
if s3Error := checkRequestAuthType(ctx, r, policy.PutBucketLifecycleAction, bucket, ""); s3Error != ErrNone {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
+23 -12
View File
@@ -17,10 +17,12 @@
package cmd
import (
"bytes"
"encoding/xml"
"errors"
"io"
"net/http"
"path"
"github.com/gorilla/mux"
xhttp "github.com/minio/minio/cmd/http"
@@ -49,6 +51,7 @@ func (api objectAPIHandlers) GetBucketNotificationHandler(w http.ResponseWriter,
vars := mux.Vars(r)
bucketName := vars["bucket"]
var config *event.Config
objAPI := api.ObjectAPI()
if objAPI == nil {
@@ -72,24 +75,31 @@ func (api objectAPIHandlers) GetBucketNotificationHandler(w http.ResponseWriter,
return
}
// Attempt to successfully load notification config.
nConfig, err := readNotificationConfig(ctx, objAPI, bucketName)
// Construct path to notification.xml for the given bucket.
configFile := path.Join(bucketConfigPrefix, bucketName, bucketNotificationConfig)
configData, err := readConfig(ctx, objAPI, configFile)
if err != nil {
// Ignore errNoSuchNotifications to comply with AWS S3.
if err != errNoSuchNotifications {
if err != errConfigNotFound {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
config = &event.Config{}
} else {
if err = xml.NewDecoder(bytes.NewReader(configData)).Decode(&config); err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
nConfig = &event.Config{}
}
config.SetRegion(globalServerConfig.GetRegion())
// If xml namespace is empty, set a default value before returning.
if nConfig.XMLNS == "" {
nConfig.XMLNS = "http://s3.amazonaws.com/doc/2006-03-01/"
if config.XMLNS == "" {
config.XMLNS = "http://s3.amazonaws.com/doc/2006-03-01/"
}
notificationBytes, err := xml.Marshal(nConfig)
notificationBytes, err := xml.Marshal(config)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
@@ -143,9 +153,10 @@ func (api objectAPIHandlers) PutBucketNotificationHandler(w http.ResponseWriter,
if event.IsEventError(err) {
apiErr = toAPIError(ctx, err)
}
writeErrorResponse(ctx, w, apiErr, r.URL, guessIsBrowserReq(r))
return
if _, ok := err.(*event.ErrARNNotFound); !ok {
writeErrorResponse(ctx, w, apiErr, r.URL, guessIsBrowserReq(r))
return
}
}
if err = saveNotificationConfig(ctx, objectAPI, bucketName, config); err != nil {
+1 -1
View File
@@ -87,7 +87,7 @@ func getRootCAs(certsCAsDir string) (*x509.CertPool, error) {
// Load all custom CA files.
for _, fi := range fis {
// Skip all directories.
if hasSuffix(fi, slashSeparator) {
if hasSuffix(fi, SlashSeparator) {
continue
}
caCert, err := ioutil.ReadFile(pathJoin(certsCAsDir, fi))
+27 -3
View File
@@ -38,10 +38,32 @@ import (
xnet "github.com/minio/minio/pkg/net"
)
func verifyObjectLayerFeatures(name string, objAPI ObjectLayer) {
if (globalAutoEncryption || GlobalKMS != nil) && !objAPI.IsEncryptionSupported() {
logger.Fatal(errInvalidArgument,
"Encryption support is requested but '%s' does not support encryption", name)
}
if strings.HasPrefix(name, "gateway") {
if GlobalGatewaySSE.IsSet() && GlobalKMS == nil {
uiErr := uiErrInvalidGWSSEEnvValue(nil).Msg("MINIO_GATEWAY_SSE set but KMS is not configured")
logger.Fatal(uiErr, "Unable to start gateway with SSE")
}
}
if globalIsCompressionEnabled && !objAPI.IsCompressionSupported() {
logger.Fatal(errInvalidArgument,
"Compression support is requested but '%s' does not support compression", name)
}
}
// Check for updates and print a notification message
func checkUpdate(mode string) {
// Its OK to ignore any errors during doUpdate() here.
if updateMsg, _, currentReleaseTime, latestReleaseTime, err := getUpdateInfo(2*time.Second, mode); err == nil {
if updateMsg == "" {
return
}
if globalInplaceUpdateDisabled {
logger.StartupMessage(updateMsg)
} else {
@@ -52,21 +74,23 @@ func checkUpdate(mode string) {
// Load logger targets based on user's configuration
func loadLoggers() {
loggerUserAgent := getUserAgent(getMinioMode())
auditEndpoint, ok := os.LookupEnv("MINIO_AUDIT_LOGGER_HTTP_ENDPOINT")
if ok {
// Enable audit HTTP logging through ENV.
logger.AddAuditTarget(http.New(auditEndpoint, NewCustomHTTPTransport()))
logger.AddAuditTarget(http.New(auditEndpoint, loggerUserAgent, NewCustomHTTPTransport()))
}
loggerEndpoint, ok := os.LookupEnv("MINIO_LOGGER_HTTP_ENDPOINT")
if ok {
// Enable HTTP logging through ENV.
logger.AddTarget(http.New(loggerEndpoint, NewCustomHTTPTransport()))
logger.AddTarget(http.New(loggerEndpoint, loggerUserAgent, NewCustomHTTPTransport()))
} else {
for _, l := range globalServerConfig.Logger.HTTP {
if l.Enabled {
// Enable http logging
logger.AddTarget(http.New(l.Endpoint, NewCustomHTTPTransport()))
logger.AddTarget(http.New(l.Endpoint, loggerUserAgent, NewCustomHTTPTransport()))
}
}
}
-53
View File
@@ -20,9 +20,7 @@ import (
"bytes"
"context"
"errors"
"fmt"
etcd "github.com/coreos/etcd/clientv3"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/hash"
)
@@ -51,38 +49,10 @@ func readConfig(ctx context.Context, objAPI ObjectLayer, configFile string) ([]b
return buffer.Bytes(), nil
}
func deleteConfigEtcd(ctx context.Context, client *etcd.Client, configFile string) error {
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
_, err := client.Delete(timeoutCtx, configFile)
if err != nil {
if err == context.DeadlineExceeded {
return fmt.Errorf("etcd setup is unreachable, please check your endpoints %s",
client.Endpoints())
}
return fmt.Errorf("unexpected error %s returned by etcd setup, please check your endpoints %s",
err, client.Endpoints())
}
return nil
}
func deleteConfig(ctx context.Context, objAPI ObjectLayer, configFile string) error {
return objAPI.DeleteObject(ctx, minioMetaBucket, configFile)
}
func saveConfigEtcd(ctx context.Context, client *etcd.Client, configFile string, data []byte) error {
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
_, err := client.Put(timeoutCtx, configFile, string(data))
if err == context.DeadlineExceeded {
return fmt.Errorf("etcd setup is unreachable, please check your endpoints %s", client.Endpoints())
} else if err != nil {
return fmt.Errorf("unexpected error %s returned by etcd setup, please check your endpoints %s", err, client.Endpoints())
}
return nil
}
func saveConfig(ctx context.Context, objAPI ObjectLayer, configFile string, data []byte) error {
hashReader, err := hash.NewReader(bytes.NewReader(data), int64(len(data)), "", getSHA256Hash(data), int64(len(data)), globalCLIContext.StrictS3Compat)
if err != nil {
@@ -93,29 +63,6 @@ func saveConfig(ctx context.Context, objAPI ObjectLayer, configFile string, data
return err
}
func readConfigEtcd(ctx context.Context, client *etcd.Client, configFile string) ([]byte, error) {
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
resp, err := client.Get(timeoutCtx, configFile)
if err != nil {
if err == context.DeadlineExceeded {
return nil, fmt.Errorf("etcd setup is unreachable, please check your endpoints %s",
client.Endpoints())
}
return nil, fmt.Errorf("unexpected error %s returned by etcd setup, please check your endpoints %s",
err, client.Endpoints())
}
if resp.Count == 0 {
return nil, errConfigNotFound
}
for _, ev := range resp.Kvs {
if string(ev.Key) == configFile {
return ev.Value, nil
}
}
return nil, errConfigNotFound
}
func checkConfig(ctx context.Context, objAPI ObjectLayer, configFile string) error {
if _, err := objAPI.GetObjectInfo(ctx, minioMetaBucket, configFile, ObjectOptions{}); err != nil {
// Treat object not found as config not found.
+33 -16
View File
@@ -30,7 +30,7 @@ import (
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/event/target"
"github.com/minio/minio/pkg/iam/policy"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/iam/validator"
xnet "github.com/minio/minio/pkg/net"
)
@@ -281,17 +281,27 @@ func (s *serverConfig) loadFromEnvs() {
}
if jwksURL, ok := os.LookupEnv("MINIO_IAM_JWKS_URL"); ok {
if u, err := xnet.ParseURL(jwksURL); err == nil {
s.OpenID.JWKS.URL = u
logger.FatalIf(s.OpenID.JWKS.PopulatePublicKey(), "Unable to populate public key from JWKS URL")
u, err := xnet.ParseURL(jwksURL)
if err != nil {
logger.FatalIf(err, "Unable to parse MINIO_IAM_JWKS_URL %s", jwksURL)
}
s.OpenID.JWKS.URL = u
}
if opaURL, ok := os.LookupEnv("MINIO_IAM_OPA_URL"); ok {
if u, err := xnet.ParseURL(opaURL); err == nil {
s.Policy.OPA.URL = u
s.Policy.OPA.AuthToken = os.Getenv("MINIO_IAM_OPA_AUTHTOKEN")
u, err := xnet.ParseURL(opaURL)
if err != nil {
logger.FatalIf(err, "Unable to parse MINIO_IAM_OPA_URL %s", opaURL)
}
opaArgs := iampolicy.OpaArgs{
URL: u,
AuthToken: os.Getenv("MINIO_IAM_OPA_AUTHTOKEN"),
Transport: NewCustomHTTPTransport(),
CloseRespFn: xhttp.DrainBody,
}
logger.FatalIf(opaArgs.Validate(), "Unable to reach MINIO_IAM_OPA_URL %s", opaURL)
s.Policy.OPA.URL = opaArgs.URL
s.Policy.OPA.AuthToken = opaArgs.AuthToken
}
}
@@ -303,7 +313,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewAMQPTarget(k, v)
t, err := target.NewAMQPTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("amqp(%s): %s", k, err.Error())
}
@@ -347,7 +357,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewMySQLTarget(k, v)
t, err := target.NewMySQLTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("mysql(%s): %s", k, err.Error())
}
@@ -380,7 +390,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewPostgreSQLTarget(k, v)
t, err := target.NewPostgreSQLTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("postgreSQL(%s): %s", k, err.Error())
}
@@ -536,7 +546,7 @@ func (s *serverConfig) loadToCachedConfigs() {
globalCacheMaxUse = cacheConf.MaxUse
}
if err := Environment.LookupKMSConfig(s.KMS); err != nil {
logger.FatalIf(err, "Unable to setup the KMS")
logger.FatalIf(err, "Unable to setup the KMS %s", s.KMS.Vault.Endpoint)
}
if !globalIsCompressionEnabled {
@@ -546,15 +556,22 @@ func (s *serverConfig) loadToCachedConfigs() {
globalIsCompressionEnabled = compressionConf.Enabled
}
if s.OpenID.JWKS.URL != nil && s.OpenID.JWKS.URL.String() != "" {
logger.FatalIf(s.OpenID.JWKS.PopulatePublicKey(),
"Unable to populate public key from JWKS URL %s", s.OpenID.JWKS.URL)
}
globalIAMValidators = getAuthValidators(s)
if s.Policy.OPA.URL != nil && s.Policy.OPA.URL.String() != "" {
globalPolicyOPA = iampolicy.NewOpa(iampolicy.OpaArgs{
opaArgs := iampolicy.OpaArgs{
URL: s.Policy.OPA.URL,
AuthToken: s.Policy.OPA.AuthToken,
Transport: NewCustomHTTPTransport(),
CloseRespFn: xhttp.DrainBody,
})
}
logger.FatalIf(opaArgs.Validate(), "Unable to reach OPA URL %s", s.Policy.OPA.URL)
globalPolicyOPA = iampolicy.NewOpa(opaArgs)
}
}
@@ -637,7 +654,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
}
for id, args := range config.Notify.AMQP {
if args.Enable {
newTarget, err := target.NewAMQPTarget(id, args)
newTarget, err := target.NewAMQPTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
@@ -696,7 +713,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.MySQL {
if args.Enable {
newTarget, err := target.NewMySQLTarget(id, args)
newTarget, err := target.NewMySQLTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
@@ -738,7 +755,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.PostgreSQL {
if args.Enable {
newTarget, err := target.NewPostgreSQLTarget(id, args)
newTarget, err := target.NewPostgreSQLTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
+3 -3
View File
@@ -185,7 +185,7 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "browser": "on", "browser": "on", "region":"us-east-1", "credential" : {"accessKey":"minio", "secretKey":"minio123"}}`, false},
// Test 11 - Test AMQP
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "amqp": { "1": { "enable": true, "url": "", "exchange": "", "routingKey": "", "exchangeType": "", "mandatory": false, "immediate": false, "durable": false, "internal": false, "noWait": false, "autoDeleted": false }}}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "amqp": { "1": { "enable": true, "url": "", "exchange": "", "routingKey": "", "exchangeType": "", "mandatory": false, "immediate": false, "durable": false, "internal": false, "noWait": false, "autoDeleted": false, "queueDir": "", "queueLimit": 0}}}}`, false},
// Test 12 - Test NATS
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "nats": { "1": { "enable": true, "address": "", "subject": "", "username": "", "password": "", "token": "", "secure": false, "pingInterval": 0, "queueDir": "", "queueLimit": 0, "streaming": { "enable": false, "clusterID": "", "async": false, "maxPubAcksInflight": 0 } } }}}`, false},
@@ -197,7 +197,7 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "redis": { "1": { "enable": true, "address": "", "password": "", "key": "" } }}}`, false},
// Test 15 - Test PostgreSQL
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "postgresql": { "1": { "enable": true, "connectionString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "" }}}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "postgresql": { "1": { "enable": true, "connectionString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "", "queueDir": "", "queueLimit": 0 }}}}`, false},
// Test 16 - Test Kafka
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "kafka": { "1": { "enable": true, "brokers": null, "topic": "", "queueDir": "", "queueLimit": 0 } }}}`, false},
@@ -206,7 +206,7 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "webhook": { "1": { "enable": true, "endpoint": "", "queueDir": "", "queueLimit": 0} }}}`, false},
// Test 18 - Test MySQL
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mysql": { "1": { "enable": true, "dsnString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "" }}}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mysql": { "1": { "enable": true, "dsnString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "", "queueDir": "", "queueLimit": 0 }}}}`, false},
// Test 19 - Test Format for MySQL
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mysql": { "1": { "enable": true, "dsnString": "", "format": "invalid", "table": "xxx", "host": "10.0.0.1", "port": "3306", "user": "abc", "password": "pqr", "database": "test1" }}}}`, false},
+1 -1
View File
@@ -2426,7 +2426,7 @@ func migrateConfigToMinioSys(objAPI ObjectLayer) (err error) {
defer func() {
if err == nil {
if globalEtcdClient != nil {
deleteConfigEtcd(context.Background(), globalEtcdClient, configFile)
deleteKeyEtcd(context.Background(), globalEtcdClient, configFile)
} else {
// Rename config.json to config.json.deprecated only upon
// success of this function.
+3 -3
View File
@@ -175,7 +175,7 @@ func TestServerConfigMigrateV2toV33(t *testing.T) {
}
defer os.RemoveAll(fsDir)
configPath := rootPath + "/" + minioConfigFile
configPath := rootPath + SlashSeparator + minioConfigFile
// Create a corrupted config file
if err := ioutil.WriteFile(configPath, []byte("{ \"version\":\"2\","), 0644); err != nil {
@@ -238,7 +238,7 @@ func TestServerConfigMigrateFaultyConfig(t *testing.T) {
defer os.RemoveAll(rootPath)
globalConfigDir = &ConfigDir{path: rootPath}
configPath := rootPath + "/" + minioConfigFile
configPath := rootPath + SlashSeparator + minioConfigFile
// Create a corrupted config file
if err := ioutil.WriteFile(configPath, []byte("{ \"version\":\"2\", \"test\":"), 0644); err != nil {
@@ -335,7 +335,7 @@ func TestServerConfigMigrateCorruptedConfig(t *testing.T) {
defer os.RemoveAll(rootPath)
globalConfigDir = &ConfigDir{path: rootPath}
configPath := rootPath + "/" + minioConfigFile
configPath := rootPath + SlashSeparator + minioConfigFile
for i := 3; i <= 17; i++ {
// Create a corrupted config file
+20
View File
@@ -23,6 +23,7 @@ import (
"path"
"runtime"
"strings"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/quick"
@@ -101,6 +102,25 @@ func (sys *ConfigSys) Load(objAPI ObjectLayer) error {
return sys.Init(objAPI)
}
// WatchConfigNASDisk - watches nas disk on periodic basis.
func (sys *ConfigSys) WatchConfigNASDisk(objAPI ObjectLayer) {
configInterval := globalRefreshIAMInterval
watchDisk := func() {
ticker := time.NewTicker(configInterval)
defer ticker.Stop()
for {
select {
case <-GlobalServiceDoneCh:
return
case <-ticker.C:
loadConfig(objAPI)
}
}
}
// Refresh configSys in background for NAS gateway.
go watchDisk()
}
// Init - initializes config system from config.json.
func (sys *ConfigSys) Init(objAPI ObjectLayer) error {
if objAPI == nil {
+2
View File
@@ -62,6 +62,8 @@ var (
errInvalidInternalIV = Error{"The internal encryption IV is malformed"}
errInvalidInternalSealAlgorithm = Error{"The internal seal algorithm is invalid and not supported"}
errMissingUpdatedKey = Error{"The key update returned no error but also no sealed key"}
)
var (
+20
View File
@@ -86,6 +86,19 @@ type KMS interface {
// referenced by the keyID. The provided context must
// match the context used to generate the sealed key.
UnsealKey(keyID string, sealedKey []byte, context Context) (key [32]byte, err error)
// UpdateKey re-wraps the sealedKey if the master key, referenced by
// `keyID`, has changed in the meantime. This usually happens when the
// KMS operator performs a key-rotation operation of the master key.
// UpdateKey fails if the provided sealedKey cannot be decrypted using
// the master key referenced by keyID.
//
// UpdateKey makes no guarantees whatsoever about whether the returned
// rotatedKey is actually different from the sealedKey. If nothing has
// changed at the KMS or if the KMS does not support updating generated
// keys this method may behave like a NOP and just return the sealedKey
// itself.
UpdateKey(keyID string, sealedKey []byte, context Context) (rotatedKey []byte, err error)
}
type masterKeyKMS struct {
@@ -126,6 +139,13 @@ func (kms *masterKeyKMS) UnsealKey(keyID string, sealedKey []byte, ctx Context)
return key, nil
}
func (kms *masterKeyKMS) UpdateKey(keyID string, sealedKey []byte, ctx Context) ([]byte, error) {
if _, err := kms.UnsealKey(keyID, sealedKey, ctx); err != nil {
return nil, err
}
return sealedKey, nil // The master key cannot update data keys -> Do nothing.
}
func (kms *masterKeyKMS) deriveKey(keyID string, context Context) (key [32]byte) {
if context == nil {
context = Context{}
+10 -1
View File
@@ -51,11 +51,20 @@ func TestMasterKeyKMS(t *testing.T) {
t.Errorf("Test %d: KMS failed to unseal the generated key: %v", i, err)
}
if err == nil && test.ShouldFail {
t.Errorf("Test %d: KMS unsealed the generated successfully but should have failed", i)
t.Errorf("Test %d: KMS unsealed the generated key successfully but should have failed", i)
}
if !test.ShouldFail && !bytes.Equal(key[:], unsealedKey[:]) {
t.Errorf("Test %d: The generated and unsealed key differ", i)
}
rotatedKey, err := kms.UpdateKey(test.UnsealKeyID, sealedKey, test.UnsealContext)
if err == nil && test.ShouldFail {
t.Errorf("Test %d: KMS updated the generated key successfully but should have failed", i)
}
if !test.ShouldFail && !bytes.Equal(rotatedKey, sealedKey[:]) {
t.Errorf("Test %d: The updated and sealed key differ", i)
}
}
}
+27
View File
@@ -250,3 +250,30 @@ func (v *vaultService) UnsealKey(keyID string, sealedKey []byte, ctx Context) (k
copy(key[:], []byte(plainKey))
return key, nil
}
// UpdateKey re-wraps the sealedKey if the master key referenced by the keyID
// has been changed by the KMS operator - i.e. the master key has been rotated.
// If the master key hasn't changed since the sealedKey has been created / updated
// it may return the same sealedKey as rotatedKey.
//
// The context must be same context as the one provided while
// generating the plaintext key / sealedKey.
func (v *vaultService) UpdateKey(keyID string, sealedKey []byte, ctx Context) (rotatedKey []byte, err error) {
var contextStream bytes.Buffer
ctx.WriteTo(&contextStream)
payload := map[string]interface{}{
"ciphertext": string(sealedKey),
"context": base64.StdEncoding.EncodeToString(contextStream.Bytes()),
}
s, err := v.client.Logical().Write(fmt.Sprintf("/transit/rewrap/%s", keyID), payload)
if err != nil {
return nil, err
}
ciphertext, ok := s.Data["ciphertext"]
if !ok {
return nil, errMissingUpdatedKey
}
rotatedKey = []byte(ciphertext.(string))
return rotatedKey, nil
}
+164
View File
@@ -0,0 +1,164 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lifecycle"
)
const (
bgLifecycleInterval = 24 * time.Hour
bgLifecycleTick = time.Hour
)
type lifecycleOps struct {
LastActivity time.Time
}
// Register to the daily objects listing
var globalLifecycleOps = &lifecycleOps{}
func getLocalBgLifecycleOpsStatus() BgLifecycleOpsStatus {
return BgLifecycleOpsStatus{
LastActivity: globalLifecycleOps.LastActivity,
}
}
// initDailyLifecycle starts the routine that receives the daily
// listing of all objects and applies any matching bucket lifecycle
// rules.
func initDailyLifecycle() {
go startDailyLifecycle()
}
func startDailyLifecycle() {
var objAPI ObjectLayer
var ctx = context.Background()
// Wait until the object API is ready
for {
objAPI = newObjectLayerFn()
if objAPI == nil {
time.Sleep(time.Second)
continue
}
break
}
// Calculate the time of the last lifecycle operation in all peers node of the cluster
computeLastLifecycleActivity := func(status []BgOpsStatus) time.Time {
var lastAct time.Time
for _, st := range status {
if st.LifecycleOps.LastActivity.After(lastAct) {
lastAct = st.LifecycleOps.LastActivity
}
}
return lastAct
}
for {
// Check if we should perform lifecycle ops based on the last lifecycle activity, sleep one hour otherwise
allLifecycleStatus := []BgOpsStatus{
{LifecycleOps: getLocalBgLifecycleOpsStatus()},
}
if globalIsDistXL {
allLifecycleStatus = append(allLifecycleStatus, globalNotificationSys.BackgroundOpsStatus()...)
}
lastAct := computeLastLifecycleActivity(allLifecycleStatus)
if !lastAct.IsZero() && time.Since(lastAct) < bgLifecycleInterval {
time.Sleep(bgLifecycleTick)
}
// Perform one lifecycle operation
err := lifecycleRound(ctx, objAPI)
switch err.(type) {
// Unable to hold a lock means there is another
// instance doing the lifecycle round round
case OperationTimedOut:
time.Sleep(bgLifecycleTick)
default:
logger.LogIf(ctx, err)
time.Sleep(time.Minute)
continue
}
}
}
func lifecycleRound(ctx context.Context, objAPI ObjectLayer) error {
zeroDuration := time.Millisecond
zeroDynamicTimeout := newDynamicTimeout(zeroDuration, zeroDuration)
// Lock to avoid concurrent lifecycle ops from other nodes
sweepLock := globalNSMutex.NewNSLock(ctx, "system", "daily-lifecycle-ops")
if err := sweepLock.GetLock(zeroDynamicTimeout); err != nil {
return err
}
defer sweepLock.Unlock()
buckets, err := objAPI.ListBuckets(ctx)
if err != nil {
return err
}
for _, bucket := range buckets {
// Check if the current bucket has a configured lifecycle policy, skip otherwise
l, ok := globalLifecycleSys.Get(bucket.Name)
if !ok {
continue
}
// Calculate the common prefix of all lifecycle rules
var prefixes []string
for _, rule := range l.Rules {
prefixes = append(prefixes, rule.Filter.Prefix)
}
commonPrefix := lcp(prefixes)
// List all objects and calculate lifecycle action based on object name & object modtime
marker := ""
for {
res, err := objAPI.ListObjects(ctx, bucket.Name, commonPrefix, marker, "", 1000)
if err != nil {
continue
}
for _, obj := range res.Objects {
// Find the action that need to be executed
action := l.ComputeAction(obj.Name, obj.ModTime)
switch action {
case lifecycle.DeleteAction:
objAPI.DeleteObject(ctx, bucket.Name, obj.Name)
default:
// Nothing
}
}
if !res.IsTruncated {
break
} else {
marker = res.NextMarker
}
}
}
return nil
}
+573
View File
@@ -0,0 +1,573 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"bytes"
"context"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"path"
"reflect"
"sync"
"time"
"github.com/djherbis/atime"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/disk"
"github.com/ncw/directio"
)
const (
// cache.json object metadata for cached objects.
cacheMetaJSONFile = "cache.json"
cacheDataFile = "part.1"
cacheMetaVersion = "1.0.0"
cacheEnvDelimiter = ";"
)
// CacheChecksumInfoV1 - carries checksums of individual blocks on disk.
type CacheChecksumInfoV1 struct {
Algorithm string `json:"algorithm"`
Blocksize int64 `json:"blocksize"`
}
// Represents the cache metadata struct
type cacheMeta struct {
Version string `json:"version"`
Stat statInfo `json:"stat"` // Stat of the current object `cache.json`.
// checksums of blocks on disk.
Checksum CacheChecksumInfoV1 `json:"checksum,omitempty"`
// Metadata map for current object.
Meta map[string]string `json:"meta,omitempty"`
}
func (m *cacheMeta) ToObjectInfo(bucket, object string) (o ObjectInfo) {
if len(m.Meta) == 0 {
m.Meta = make(map[string]string)
m.Stat.ModTime = timeSentinel
}
o = ObjectInfo{
Bucket: bucket,
Name: object,
}
// We set file info only if its valid.
o.ModTime = m.Stat.ModTime
o.Size = m.Stat.Size
o.ETag = extractETag(m.Meta)
o.ContentType = m.Meta["content-type"]
o.ContentEncoding = m.Meta["content-encoding"]
if storageClass, ok := m.Meta[amzStorageClass]; ok {
o.StorageClass = storageClass
} else {
o.StorageClass = globalMinioDefaultStorageClass
}
var (
t time.Time
e error
)
if exp, ok := m.Meta["expires"]; ok {
if t, e = time.Parse(http.TimeFormat, exp); e == nil {
o.Expires = t.UTC()
}
}
// etag/md5Sum has already been extracted. We need to
// remove to avoid it from appearing as part of user-defined metadata
o.UserDefined = cleanMetadata(m.Meta)
return o
}
// represents disk cache struct
type diskCache struct {
dir string // caching directory
maxDiskUsagePct int // max usage in %
expiry int // cache expiry in days
// mark false if drive is offline
online bool
// mutex to protect updates to online variable
onlineMutex *sync.RWMutex
// purge() listens on this channel to start the cache-purge process
purgeChan chan struct{}
pool sync.Pool
}
// Inits the disk cache dir if it is not initialized already.
func newdiskCache(dir string, expiry int, maxDiskUsagePct int) (*diskCache, error) {
if err := os.MkdirAll(dir, 0777); err != nil {
return nil, fmt.Errorf("Unable to initialize '%s' dir, %s", dir, err)
}
if expiry == 0 {
expiry = globalCacheExpiry
}
cache := diskCache{
dir: dir,
expiry: expiry,
maxDiskUsagePct: maxDiskUsagePct,
purgeChan: make(chan struct{}),
online: true,
onlineMutex: &sync.RWMutex{},
pool: sync.Pool{
New: func() interface{} {
b := directio.AlignedBlock(int(cacheBlkSize))
return &b
},
},
}
return &cache, nil
}
// Returns if the disk usage is low.
// Disk usage is low if usage is < 80% of cacheMaxDiskUsagePct
// Ex. for a 100GB disk, if maxUsage is configured as 70% then cacheMaxDiskUsagePct is 70G
// hence disk usage is low if the disk usage is less than 56G (because 80% of 70G is 56G)
func (c *diskCache) diskUsageLow() bool {
minUsage := c.maxDiskUsagePct * 80 / 100
di, err := disk.GetInfo(c.dir)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", c.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return false
}
usedPercent := (di.Total - di.Free) * 100 / di.Total
return int(usedPercent) < minUsage
}
// Return if the disk usage is high.
// Disk usage is high if disk used is > cacheMaxDiskUsagePct
func (c *diskCache) diskUsageHigh() bool {
di, err := disk.GetInfo(c.dir)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", c.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return true
}
usedPercent := (di.Total - di.Free) * 100 / di.Total
return int(usedPercent) > c.maxDiskUsagePct
}
// Returns if size space can be allocated without exceeding
// max disk usable for caching
func (c *diskCache) diskAvailable(size int64) bool {
di, err := disk.GetInfo(c.dir)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", c.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return false
}
usedPercent := (di.Total - (di.Free - uint64(size))) * 100 / di.Total
return int(usedPercent) < c.maxDiskUsagePct
}
// Purge cache entries that were not accessed.
func (c *diskCache) purge() {
ctx := context.Background()
for {
olderThan := c.expiry
for !c.diskUsageLow() {
// delete unaccessed objects older than expiry duration
expiry := UTCNow().AddDate(0, 0, -1*olderThan)
olderThan /= 2
if olderThan < 1 {
break
}
deletedCount := 0
objDirs, err := ioutil.ReadDir(c.dir)
if err != nil {
log.Fatal(err)
}
for _, obj := range objDirs {
if obj.Name() == minioMetaBucket {
continue
}
// stat entry to get atime
var fi os.FileInfo
fi, err := os.Stat(pathJoin(c.dir, obj.Name(), cacheDataFile))
if err != nil {
continue
}
objInfo, err := c.statCache(ctx, pathJoin(c.dir, obj.Name()))
if err != nil {
// delete any partially filled cache entry left behind.
removeAll(pathJoin(c.dir, obj.Name()))
continue
}
cc := cacheControlOpts(objInfo)
if atime.Get(fi).Before(expiry) ||
cc.isStale(objInfo.ModTime) {
if err = removeAll(pathJoin(c.dir, obj.Name())); err != nil {
logger.LogIf(ctx, err)
}
deletedCount++
// break early if sufficient disk space reclaimed.
if !c.diskUsageLow() {
break
}
}
}
if deletedCount == 0 {
break
}
}
lastRunTime := time.Now()
for {
<-c.purgeChan
timeElapsed := time.Since(lastRunTime)
if timeElapsed > time.Hour {
break
}
}
}
}
// sets cache drive status
func (c *diskCache) setOnline(status bool) {
c.onlineMutex.Lock()
c.online = status
c.onlineMutex.Unlock()
}
// returns true if cache drive is online
func (c *diskCache) IsOnline() bool {
c.onlineMutex.RLock()
defer c.onlineMutex.RUnlock()
return c.online
}
// Stat returns ObjectInfo from disk cache
func (c *diskCache) Stat(ctx context.Context, bucket, object string) (oi ObjectInfo, err error) {
cacheObjPath := getCacheSHADir(c.dir, bucket, object)
oi, err = c.statCache(ctx, cacheObjPath)
if err != nil {
return
}
oi.Bucket = bucket
oi.Name = object
return
}
// statCache is a convenience function for purge() to get ObjectInfo for cached object
func (c *diskCache) statCache(ctx context.Context, cacheObjPath string) (oi ObjectInfo, e error) {
// Stat the file to get file size.
metaPath := path.Join(cacheObjPath, cacheMetaJSONFile)
f, err := os.Open(metaPath)
if err != nil {
return oi, err
}
defer f.Close()
meta := &cacheMeta{Version: cacheMetaVersion}
if err := jsonLoad(f, meta); err != nil {
return oi, err
}
fi, err := os.Stat(pathJoin(cacheObjPath, cacheDataFile))
if err != nil {
return oi, err
}
meta.Stat.ModTime = atime.Get(fi)
return meta.ToObjectInfo("", ""), nil
}
// saves object metadata to disk cache
func (c *diskCache) saveMetadata(ctx context.Context, bucket, object string, meta map[string]string, actualSize int64) error {
fileName := getCacheSHADir(c.dir, bucket, object)
metaPath := pathJoin(fileName, cacheMetaJSONFile)
f, err := os.Create(metaPath)
if err != nil {
return err
}
defer f.Close()
m := cacheMeta{Meta: meta, Version: cacheMetaVersion}
m.Stat.Size = actualSize
m.Stat.ModTime = UTCNow()
m.Checksum = CacheChecksumInfoV1{Algorithm: HighwayHash256S.String(), Blocksize: cacheBlkSize}
jsonData, err := json.Marshal(m)
if err != nil {
return err
}
_, err = f.Write(jsonData)
return err
}
// Backend metadata could have changed through server side copy - reset cache metadata if that is the case
func (c *diskCache) updateMetadataIfChanged(ctx context.Context, bucket, object string, bkObjectInfo, cacheObjInfo ObjectInfo) error {
if !reflect.DeepEqual(bkObjectInfo.UserDefined, cacheObjInfo.UserDefined) ||
bkObjectInfo.ETag != cacheObjInfo.ETag ||
bkObjectInfo.ContentType != cacheObjInfo.ContentType ||
bkObjectInfo.Expires != cacheObjInfo.Expires {
return c.saveMetadata(ctx, bucket, object, getMetadata(bkObjectInfo), bkObjectInfo.Size)
}
return nil
}
func getCacheSHADir(dir, bucket, object string) string {
return path.Join(dir, getSHA256Hash([]byte(path.Join(bucket, object))))
}
// Cache data to disk with bitrot checksum added for each block of 1MB
func (c *diskCache) bitrotWriteToCache(ctx context.Context, cachePath string, reader io.Reader, size int64) (int64, error) {
if err := os.MkdirAll(cachePath, 0777); err != nil {
return 0, err
}
bufSize := int64(readSizeV1)
if size > 0 && bufSize > size {
bufSize = size
}
filePath := path.Join(cachePath, cacheDataFile)
if filePath == "" || reader == nil {
return 0, errInvalidArgument
}
if err := checkPathLength(filePath); err != nil {
return 0, err
}
f, err := os.Create(filePath)
if err != nil {
return 0, osErrToFSFileErr(err)
}
defer f.Close()
var bytesWritten int64
h := HighwayHash256S.New()
bufp := c.pool.Get().(*[]byte)
defer c.pool.Put(bufp)
for {
n, err := io.ReadFull(reader, *bufp)
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF && err != io.ErrClosedPipe {
return 0, err
}
eof := err == io.EOF || err == io.ErrUnexpectedEOF || err == io.ErrClosedPipe
if n == 0 && size != 0 {
// Reached EOF, nothing more to be done.
break
}
h.Reset()
if _, err := h.Write((*bufp)[:n]); err != nil {
return 0, err
}
hashBytes := h.Sum(nil)
if _, err = f.Write(hashBytes); err != nil {
return 0, err
}
if _, err = f.Write((*bufp)[:n]); err != nil {
return 0, err
}
bytesWritten += int64(n)
if eof {
break
}
}
return bytesWritten, nil
}
// Caches the object to disk
func (c *diskCache) Put(ctx context.Context, bucket, object string, data io.Reader, size int64, opts ObjectOptions) error {
if c.diskUsageHigh() {
select {
case c.purgeChan <- struct{}{}:
default:
}
return errDiskFull
}
if !c.diskAvailable(size) {
return errDiskFull
}
cachePath := getCacheSHADir(c.dir, bucket, object)
if err := os.MkdirAll(cachePath, 0777); err != nil {
return err
}
bufSize := int64(readSizeV1)
if size > 0 && bufSize > size {
bufSize = size
}
n, err := c.bitrotWriteToCache(ctx, cachePath, data, size)
if IsErr(err, baseErrs...) {
c.setOnline(false)
}
if err != nil {
return err
}
return c.saveMetadata(ctx, bucket, object, opts.UserDefined, n)
}
// checks streaming bitrot checksum of cached object before returning data
func (c *diskCache) bitrotReadFromCache(ctx context.Context, filePath string, offset, length int64, writer io.Writer) error {
h := HighwayHash256S.New()
checksumHash := make([]byte, h.Size())
startBlock := offset / cacheBlkSize
endBlock := (offset + length) / cacheBlkSize
// get block start offset
var blockStartOffset int64
if startBlock > 0 {
blockStartOffset = (cacheBlkSize + int64(h.Size())) * startBlock
}
tillLength := (cacheBlkSize + int64(h.Size())) * (endBlock - startBlock + 1)
// Start offset cannot be negative.
if offset < 0 {
logger.LogIf(ctx, errUnexpected)
return errUnexpected
}
// Writer cannot be nil.
if writer == nil {
logger.LogIf(ctx, errUnexpected)
return errUnexpected
}
var blockOffset, blockLength int64
rc, err := readCacheFileStream(filePath, blockStartOffset, tillLength)
if err != nil {
return err
}
bufp := c.pool.Get().(*[]byte)
defer c.pool.Put(bufp)
for block := startBlock; block <= endBlock; block++ {
switch {
case startBlock == endBlock:
blockOffset = offset % cacheBlkSize
blockLength = length
case block == startBlock:
blockOffset = offset % cacheBlkSize
blockLength = cacheBlkSize - blockOffset
case block == endBlock:
blockOffset = 0
blockLength = (offset + length) % cacheBlkSize
default:
blockOffset = 0
blockLength = cacheBlkSize
}
if blockLength == 0 {
break
}
if _, err := io.ReadFull(rc, checksumHash); err != nil {
return err
}
h.Reset()
n, err := io.ReadFull(rc, *bufp)
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
logger.LogIf(ctx, err)
return err
}
eof := err == io.EOF || err == io.ErrUnexpectedEOF
if n == 0 && length != 0 {
// Reached EOF, nothing more to be done.
break
}
if _, e := h.Write((*bufp)[:n]); e != nil {
return e
}
hashBytes := h.Sum(nil)
if !bytes.Equal(hashBytes, checksumHash) {
err = HashMismatchError{hex.EncodeToString(checksumHash), hex.EncodeToString(hashBytes)}
logger.LogIf(context.Background(), err)
return err
}
if _, err := io.Copy(writer, bytes.NewReader((*bufp)[blockOffset:blockOffset+blockLength])); err != nil {
if err != io.ErrClosedPipe {
logger.LogIf(ctx, err)
}
return err
}
if eof {
break
}
}
return nil
}
// Get returns ObjectInfo and reader for object from disk cache
func (c *diskCache) Get(ctx context.Context, bucket, object string, rs *HTTPRangeSpec, h http.Header, opts ObjectOptions) (gr *GetObjectReader, err error) {
var objInfo ObjectInfo
cacheObjPath := getCacheSHADir(c.dir, bucket, object)
if objInfo, err = c.statCache(ctx, cacheObjPath); err != nil {
return nil, toObjectErr(err, bucket, object)
}
var nsUnlocker = func() {}
// For a directory, we need to send an reader that returns no bytes.
if hasSuffix(object, SlashSeparator) {
// The lock taken above is released when
// objReader.Close() is called by the caller.
return NewGetObjectReaderFromReader(bytes.NewBuffer(nil), objInfo, opts.CheckCopyPrecondFn, nsUnlocker)
}
fn, off, length, nErr := NewGetObjectReader(rs, objInfo, opts.CheckCopyPrecondFn, nsUnlocker)
if nErr != nil {
return nil, nErr
}
filePath := path.Join(cacheObjPath, cacheDataFile)
pr, pw := io.Pipe()
go func() {
pw.CloseWithError(c.bitrotReadFromCache(ctx, filePath, off, length, pw))
}()
// Cleanup function to cause the go routine above to exit, in
// case of incomplete read.
pipeCloser := func() { pr.Close() }
return fn(pr, h, opts.CheckCopyPrecondFn, pipeCloser)
}
// Deletes the cached object
func (c *diskCache) Delete(ctx context.Context, bucket, object string) (err error) {
cachePath := getCacheSHADir(c.dir, bucket, object)
return removeAll(cachePath)
}
// convenience function to check if object is cached on this diskCache
func (c *diskCache) Exists(ctx context.Context, bucket, object string) bool {
if _, err := os.Stat(getCacheSHADir(c.dir, bucket, object)); err != nil {
return false
}
return true
}
+1 -1
View File
@@ -109,7 +109,7 @@ func parseCacheExcludes(excludes []string) ([]string, error) {
if len(e) == 0 {
return nil, uiErrInvalidCacheExcludesValue(nil).Msg("cache exclude path (%s) cannot be empty", e)
}
if hasPrefix(e, slashSeparator) {
if hasPrefix(e, SlashSeparator) {
return nil, uiErrInvalidCacheExcludesValue(nil).Msg("cache exclude pattern (%s) cannot start with / as prefix", e)
}
}
-537
View File
@@ -1,537 +0,0 @@
/*
* MinIO Cloud Storage, (C) 2018 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"os"
"path"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/disk"
"github.com/minio/minio/pkg/lock"
)
const (
// cache.json object metadata for cached objects.
cacheMetaJSONFile = "cache.json"
cacheEnvDelimiter = ";"
)
// cacheFSObjects implements the cache backend operations.
type cacheFSObjects struct {
*FSObjects
// caching drive path (from cache "drives" in config.json)
dir string
// expiry in days specified in config.json
expiry int
// max disk usage pct
maxDiskUsagePct int
// purge() listens on this channel to start the cache-purge process
purgeChan chan struct{}
// mark false if drive is offline
online bool
// mutex to protect updates to online variable
onlineMutex *sync.RWMutex
}
// Inits the cache directory if it is not init'ed already.
// Initializing implies creation of new FS Object layer.
func newCacheFSObjects(dir string, expiry int, maxDiskUsagePct int) (*cacheFSObjects, error) {
// Assign a new UUID for FS minio mode. Each server instance
// gets its own UUID for temporary file transaction.
fsUUID := mustGetUUID()
// Initialize meta volume, if volume already exists ignores it.
if err := initMetaVolumeFS(dir, fsUUID); err != nil {
return nil, fmt.Errorf("Unable to initialize '.minio.sys' meta volume, %s", err)
}
trashPath := pathJoin(dir, minioMetaBucket, cacheTrashDir)
if err := os.MkdirAll(trashPath, 0777); err != nil {
return nil, err
}
if expiry == 0 {
expiry = globalCacheExpiry
}
// Initialize fs objects.
fsObjects := &FSObjects{
fsPath: dir,
metaJSONFile: cacheMetaJSONFile,
fsUUID: fsUUID,
rwPool: &fsIOPool{
readersMap: make(map[string]*lock.RLockedFile),
},
nsMutex: newNSLock(false),
listPool: NewTreeWalkPool(globalLookupTimeout),
appendFileMap: make(map[string]*fsAppendFile),
}
go fsObjects.cleanupStaleMultipartUploads(context.Background(), GlobalMultipartCleanupInterval, GlobalMultipartExpiry, GlobalServiceDoneCh)
cacheFS := cacheFSObjects{
FSObjects: fsObjects,
dir: dir,
expiry: expiry,
maxDiskUsagePct: maxDiskUsagePct,
purgeChan: make(chan struct{}),
online: true,
onlineMutex: &sync.RWMutex{},
}
return &cacheFS, nil
}
// Returns if the disk usage is low.
// Disk usage is low if usage is < 80% of cacheMaxDiskUsagePct
// Ex. for a 100GB disk, if maxUsage is configured as 70% then cacheMaxDiskUsagePct is 70G
// hence disk usage is low if the disk usage is less than 56G (because 80% of 70G is 56G)
func (cfs *cacheFSObjects) diskUsageLow() bool {
minUsage := cfs.maxDiskUsagePct * 80 / 100
di, err := disk.GetInfo(cfs.dir)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", cfs.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return false
}
usedPercent := (di.Total - di.Free) * 100 / di.Total
return int(usedPercent) < minUsage
}
// Return if the disk usage is high.
// Disk usage is high if disk used is > cacheMaxDiskUsagePct
func (cfs *cacheFSObjects) diskUsageHigh() bool {
di, err := disk.GetInfo(cfs.dir)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", cfs.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return true
}
usedPercent := (di.Total - di.Free) * 100 / di.Total
return int(usedPercent) > cfs.maxDiskUsagePct
}
// Returns if size space can be allocated without exceeding
// max disk usable for caching
func (cfs *cacheFSObjects) diskAvailable(size int64) bool {
di, err := disk.GetInfo(cfs.dir)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("cachePath", cfs.dir)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return false
}
usedPercent := (di.Total - (di.Free - uint64(size))) * 100 / di.Total
return int(usedPercent) < cfs.maxDiskUsagePct
}
// purges all content marked trash from the cache.
func (cfs *cacheFSObjects) purgeTrash() {
ticker := time.NewTicker(time.Minute * cacheCleanupInterval)
defer ticker.Stop()
for {
select {
case <-GlobalServiceDoneCh:
return
case <-ticker.C:
trashPath := path.Join(cfs.fsPath, minioMetaBucket, cacheTrashDir)
entries, err := readDir(trashPath)
if err != nil {
return
}
for _, entry := range entries {
ctx := logger.SetReqInfo(context.Background(), &logger.ReqInfo{})
fi, err := fsStatVolume(ctx, pathJoin(trashPath, entry))
if err != nil {
continue
}
dir := path.Join(trashPath, fi.Name())
// Delete all expired cache content.
fsRemoveAll(ctx, dir)
}
}
}
}
// Purge cache entries that were not accessed.
func (cfs *cacheFSObjects) purge() {
delimiter := slashSeparator
maxKeys := 1000
ctx := context.Background()
for {
olderThan := cfs.expiry
for !cfs.diskUsageLow() {
// delete unaccessed objects older than expiry duration
expiry := UTCNow().AddDate(0, 0, -1*olderThan)
olderThan /= 2
if olderThan < 1 {
break
}
deletedCount := 0
buckets, err := cfs.ListBuckets(ctx)
if err != nil {
logger.LogIf(ctx, err)
}
// Reset cache online status if drive was offline earlier.
if !cfs.IsOnline() {
cfs.setOnline(true)
}
for _, bucket := range buckets {
var continuationToken string
var marker string
for {
objects, err := cfs.ListObjects(ctx, bucket.Name, marker, continuationToken, delimiter, maxKeys)
if err != nil {
break
}
if !objects.IsTruncated {
break
}
marker = objects.NextMarker
for _, object := range objects.Objects {
// purge objects that qualify because of cache-control directives or
// past cache expiry duration.
if !filterFromCache(object.UserDefined) ||
!isStaleCache(object) ||
object.AccTime.After(expiry) {
continue
}
if err = cfs.DeleteObject(ctx, bucket.Name, object.Name); err != nil {
logger.LogIf(ctx, err)
continue
}
deletedCount++
}
}
}
if deletedCount == 0 {
// to avoid a busy loop
time.Sleep(time.Minute * 30)
}
}
<-cfs.purgeChan
}
}
// sets cache drive status
func (cfs *cacheFSObjects) setOnline(status bool) {
cfs.onlineMutex.Lock()
cfs.online = status
cfs.onlineMutex.Unlock()
}
// returns true if cache drive is online
func (cfs *cacheFSObjects) IsOnline() bool {
cfs.onlineMutex.RLock()
defer cfs.onlineMutex.RUnlock()
return cfs.online
}
// Caches the object to disk
func (cfs *cacheFSObjects) Put(ctx context.Context, bucket, object string, data *PutObjReader, opts ObjectOptions) error {
if cfs.diskUsageHigh() {
select {
case cfs.purgeChan <- struct{}{}:
default:
}
return errDiskFull
}
if !cfs.diskAvailable(data.Size()) {
return errDiskFull
}
if _, err := cfs.GetBucketInfo(ctx, bucket); err != nil {
pErr := cfs.MakeBucketWithLocation(ctx, bucket, "")
if pErr != nil {
return pErr
}
}
_, err := cfs.PutObject(ctx, bucket, object, data, opts)
// if err is due to disk being offline , mark cache drive as offline
if IsErr(err, baseErrs...) {
cfs.setOnline(false)
}
return err
}
// Returns the handle for the cached object
func (cfs *cacheFSObjects) Get(ctx context.Context, bucket, object string, startOffset int64, length int64, writer io.Writer, etag string, opts ObjectOptions) (err error) {
return cfs.GetObject(ctx, bucket, object, startOffset, length, writer, etag, opts)
}
// Deletes the cached object
func (cfs *cacheFSObjects) Delete(ctx context.Context, bucket, object string) (err error) {
return cfs.DeleteObject(ctx, bucket, object)
}
// convenience function to check if object is cached on this cacheFSObjects
func (cfs *cacheFSObjects) Exists(ctx context.Context, bucket, object string) bool {
_, err := cfs.GetObjectInfo(ctx, bucket, object, ObjectOptions{})
return err == nil
}
// Identical to fs PutObject operation except that it uses ETag in metadata
// headers.
func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object string, r *PutObjReader, opts ObjectOptions) (objInfo ObjectInfo, retErr error) {
data := r.Reader
fs := cfs.FSObjects
// Lock the object.
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetLock(globalObjectTimeout); err != nil {
return objInfo, err
}
defer objectLock.Unlock()
// No metadata is set, allocate a new one.
meta := make(map[string]string)
for k, v := range opts.UserDefined {
meta[k] = v
}
var err error
// Validate if bucket name is valid and exists.
if _, err = fs.statBucketDir(ctx, bucket); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket)
}
fsMeta := newFSMetaV1()
fsMeta.Meta = meta
// This is a special case with size as '0' and object ends
// with a slash separator, we treat it like a valid operation
// and return success.
if isObjectDir(object, data.Size()) {
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
var fi os.FileInfo
if fi, err = fsStatDir(ctx, pathJoin(fs.fsPath, bucket, object)); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
if err = checkPutObjectArgs(ctx, bucket, object, fs, data.Size()); err != nil {
return ObjectInfo{}, err
}
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
// Validate input data size and it can never be less than zero.
if data.Size() < -1 {
logger.LogIf(ctx, errInvalidArgument)
return ObjectInfo{}, errInvalidArgument
}
var wlk *lock.LockedFile
if bucket != minioMetaBucket {
bucketMetaDir := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix)
fsMetaPath := pathJoin(bucketMetaDir, bucket, object, fs.metaJSONFile)
wlk, err = fs.rwPool.Create(fsMetaPath)
if err != nil {
logger.LogIf(ctx, err)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
// This close will allow for locks to be synchronized on `fs.json`.
defer wlk.Close()
defer func() {
// Remove meta file when PutObject encounters any error
if retErr != nil {
tmpDir := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID)
fsRemoveMeta(ctx, bucketMetaDir, fsMetaPath, tmpDir)
}
}()
}
// Uploaded object will first be written to the temporary location which will eventually
// be renamed to the actual location. It is first written to the temporary location
// so that cleaning it up will be easy if the server goes down.
tempObj := mustGetUUID()
// Allocate a buffer to Read() from request body
bufSize := int64(readSizeV1)
if size := data.Size(); size > 0 && bufSize > size {
bufSize = size
}
buf := make([]byte, int(bufSize))
fsTmpObjPath := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID, tempObj)
bytesWritten, err := fsCreateFile(ctx, fsTmpObjPath, data, buf, data.Size())
if err != nil {
fsRemoveFile(ctx, fsTmpObjPath)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
if fsMeta.Meta["etag"] == "" {
fsMeta.Meta["etag"] = r.MD5CurrentHexString()
}
// Should return IncompleteBody{} error when reader has fewer
// bytes than specified in request header.
if bytesWritten < data.Size() {
fsRemoveFile(ctx, fsTmpObjPath)
return ObjectInfo{}, IncompleteBody{}
}
// Delete the temporary object in the case of a
// failure. If PutObject succeeds, then there would be
// nothing to delete.
defer fsRemoveFile(ctx, fsTmpObjPath)
// Entire object was written to the temp location, now it's safe to rename it to the actual location.
fsNSObjPath := pathJoin(fs.fsPath, bucket, object)
if err = fsRenameFile(ctx, fsTmpObjPath, fsNSObjPath); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
if bucket != minioMetaBucket {
// Write FS metadata after a successful namespace operation.
if _, err = fsMeta.WriteTo(wlk); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
}
// Stat the file to fetch timestamp, size.
fi, err := fsStatFile(ctx, pathJoin(fs.fsPath, bucket, object))
if err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
// Success.
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
// Implements S3 compatible initiate multipart API. Operation here is identical
// to fs backend implementation - with the exception that cache FS uses the uploadID
// generated on the backend
func (cfs *cacheFSObjects) NewMultipartUpload(ctx context.Context, bucket, object string, uploadID string, opts ObjectOptions) (string, error) {
if cfs.diskUsageHigh() {
select {
case cfs.purgeChan <- struct{}{}:
default:
}
return "", errDiskFull
}
if _, err := cfs.GetBucketInfo(ctx, bucket); err != nil {
pErr := cfs.MakeBucketWithLocation(ctx, bucket, "")
if pErr != nil {
return "", pErr
}
}
fs := cfs.FSObjects
if err := checkNewMultipartArgs(ctx, bucket, object, fs); err != nil {
return "", toObjectErr(err, bucket)
}
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
return "", toObjectErr(err, bucket)
}
uploadIDDir := fs.getUploadIDDir(bucket, object, uploadID)
err := mkdirAll(uploadIDDir, 0755)
if err != nil {
logger.LogIf(ctx, err)
return "", err
}
// Initialize fs.json values.
fsMeta := newFSMetaV1()
fsMeta.Meta = opts.UserDefined
fsMetaBytes, err := json.Marshal(fsMeta)
if err != nil {
logger.LogIf(ctx, err)
return "", err
}
if err = ioutil.WriteFile(pathJoin(uploadIDDir, fs.metaJSONFile), fsMetaBytes, 0644); err != nil {
logger.LogIf(ctx, err)
return "", err
}
return uploadID, nil
}
// moveBucketToTrash clears cacheFSObjects of bucket contents and moves it to trash folder.
func (cfs *cacheFSObjects) moveBucketToTrash(ctx context.Context, bucket string) (err error) {
fs := cfs.FSObjects
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if err = bucketLock.GetLock(globalObjectTimeout); err != nil {
return err
}
defer bucketLock.Unlock()
bucketDir, err := fs.getBucketDir(ctx, bucket)
if err != nil {
return toObjectErr(err, bucket)
}
trashPath := pathJoin(cfs.fsPath, minioMetaBucket, cacheTrashDir)
expiredDir := path.Join(trashPath, bucket)
// Attempt to move regular bucket to expired directory.
if err = fsRenameDir(bucketDir, expiredDir); err != nil {
logger.LogIf(ctx, err)
return toObjectErr(err, bucket)
}
// Cleanup all the bucket metadata.
ominioMetadataBucketDir := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix, bucket)
nminioMetadataBucketDir := pathJoin(trashPath, MustGetUUID())
logger.LogIf(ctx, fsRenameDir(ominioMetadataBucketDir, nminioMetadataBucketDir))
return nil
}
// Removes a directory only if its empty, handles long
// paths for windows automatically.
func fsRenameDir(dirPath, newPath string) (err error) {
if dirPath == "" || newPath == "" {
return errInvalidArgument
}
if err = checkPathLength(dirPath); err != nil {
return err
}
if err = checkPathLength(newPath); err != nil {
return err
}
if err = os.Rename(dirPath, newPath); err != nil {
if os.IsNotExist(err) {
return errVolumeNotFound
} else if isSysErrNotEmpty(err) {
return errVolumeNotEmpty
}
return err
}
return nil
}
+170
View File
@@ -0,0 +1,170 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"io"
"os"
"strconv"
"strings"
"time"
"github.com/minio/minio/cmd/crypto"
)
type cacheControl struct {
expiry time.Time
maxAge int
sMaxAge int
minFresh int
maxStale int
}
func (c cacheControl) isEmpty() bool {
return c == cacheControl{}
}
func (c cacheControl) isStale(modTime time.Time) bool {
if c.isEmpty() {
return false
}
now := time.Now()
if c.sMaxAge > 0 && c.sMaxAge < int(now.Sub(modTime).Seconds()) {
return true
}
if c.maxAge > 0 && c.maxAge < int(now.Sub(modTime).Seconds()) {
return true
}
if !c.expiry.Equal(time.Time{}) && c.expiry.Before(time.Now().Add(time.Duration(c.maxStale))) {
return true
}
if c.minFresh > 0 && c.minFresh <= int(now.Sub(modTime).Seconds()) {
return true
}
return false
}
// returns struct with cache-control settings from user metadata.
func cacheControlOpts(o ObjectInfo) (c cacheControl) {
m := o.UserDefined
if o.Expires != timeSentinel {
c.expiry = o.Expires
}
var headerVal string
for k, v := range m {
if strings.ToLower(k) == "cache-control" {
headerVal = v
}
}
if headerVal == "" {
return
}
headerVal = strings.ToLower(headerVal)
headerVal = strings.TrimSpace(headerVal)
vals := strings.Split(headerVal, ",")
for _, val := range vals {
val = strings.TrimSpace(val)
p := strings.Split(val, "=")
if len(p) != 2 {
continue
}
if p[0] == "max-age" ||
p[0] == "s-maxage" ||
p[0] == "min-fresh" ||
p[0] == "max-stale" {
i, err := strconv.Atoi(p[1])
if err != nil {
return cacheControl{}
}
if p[0] == "max-age" {
c.maxAge = i
}
if p[0] == "s-maxage" {
c.sMaxAge = i
}
if p[0] == "min-fresh" {
c.minFresh = i
}
if p[0] == "max-stale" {
c.maxStale = i
}
}
}
return c
}
// backendDownError returns true if err is due to backend failure or faulty disk if in server mode
func backendDownError(err error) bool {
_, backendDown := err.(BackendDown)
return backendDown || IsErr(err, baseErrs...)
}
// IsCacheable returns if the object should be saved in the cache.
func (o ObjectInfo) IsCacheable() bool {
return !crypto.IsEncrypted(o.UserDefined)
}
// reads file cached on disk from offset upto length
func readCacheFileStream(filePath string, offset, length int64) (io.ReadCloser, error) {
if filePath == "" || offset < 0 {
return nil, errInvalidArgument
}
if err := checkPathLength(filePath); err != nil {
return nil, err
}
fr, err := os.Open(filePath)
if err != nil {
return nil, osErrToFSFileErr(err)
}
// Stat to get the size of the file at path.
st, err := fr.Stat()
if err != nil {
err = osErrToFSFileErr(err)
return nil, err
}
// Verify if its not a regular file, since subsequent Seek is undefined.
if !st.Mode().IsRegular() {
return nil, errIsNotRegular
}
if err = os.Chtimes(filePath, time.Now(), st.ModTime()); err != nil {
return nil, err
}
// Seek to the requested offset.
if offset > 0 {
_, err = fr.Seek(offset, io.SeekStart)
if err != nil {
return nil, err
}
}
return struct {
io.Reader
io.Closer
}{Reader: io.LimitReader(fr, length), Closer: fr}, nil
}
+60
View File
@@ -0,0 +1,60 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"net/http"
"reflect"
"testing"
"time"
)
func TestGetCacheControlOpts(t *testing.T) {
expiry, _ := time.Parse(http.TimeFormat, "Wed, 21 Oct 2015 07:28:00 GMT")
testCases := []struct {
cacheControlHeaderVal string
expiryHeaderVal time.Time
expectedCacheControl cacheControl
expectedErr bool
}{
{"", timeSentinel, cacheControl{}, false},
{"max-age=2592000, public", timeSentinel, cacheControl{maxAge: 2592000, sMaxAge: 0, minFresh: 0, expiry: time.Time{}}, false},
{"max-age=2592000, no-store", timeSentinel, cacheControl{maxAge: 2592000, sMaxAge: 0, minFresh: 0, expiry: time.Time{}}, false},
{"must-revalidate, max-age=600", timeSentinel, cacheControl{maxAge: 600, sMaxAge: 0, minFresh: 0, expiry: time.Time{}}, false},
{"s-maxAge=2500, max-age=600", timeSentinel, cacheControl{maxAge: 600, sMaxAge: 2500, minFresh: 0, expiry: time.Time{}}, false},
{"s-maxAge=2500, max-age=600", expiry, cacheControl{maxAge: 600, sMaxAge: 2500, minFresh: 0, expiry: time.Date(2015, time.October, 21, 07, 28, 00, 00, time.UTC)}, false},
{"s-maxAge=2500, max-age=600s", timeSentinel, cacheControl{maxAge: 600, sMaxAge: 2500, minFresh: 0, expiry: time.Time{}}, true},
}
var m map[string]string
for i, testCase := range testCases {
m = make(map[string]string)
m["cache-control"] = testCase.cacheControlHeaderVal
if testCase.expiryHeaderVal != timeSentinel {
m["expires"] = testCase.expiryHeaderVal.String()
}
c := cacheControlOpts(ObjectInfo{UserDefined: m, Expires: testCase.expiryHeaderVal})
if testCase.expectedErr && (c != cacheControl{}) {
t.Errorf("expected err for case %d", i)
}
if !testCase.expectedErr && !reflect.DeepEqual(c, testCase.expectedCacheControl) {
t.Errorf("expected %v got %v for case %d", testCase.expectedCacheControl, c, i)
}
}
}
+284 -898
View File
File diff suppressed because it is too large Load Diff
+66 -113
View File
@@ -1,5 +1,5 @@
/*
* MinIO Cloud Storage, (C) 2018 MinIO, Inc.
* MinIO Cloud Storage, (C) 2018,2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -19,34 +19,52 @@ package cmd
import (
"bytes"
"context"
"reflect"
"io"
"net/http"
"testing"
"time"
"github.com/minio/minio/pkg/hash"
)
// Initialize cache FS objects.
func initCacheFSObjects(disk string, cacheMaxUse int) (*cacheFSObjects, error) {
return newCacheFSObjects(disk, globalCacheExpiry, cacheMaxUse)
// Initialize cache objects.
func initCacheObjects(disk string, cacheMaxUse int) (*diskCache, error) {
return newdiskCache(disk, globalCacheExpiry, cacheMaxUse)
}
// inits diskCache struct for nDisks
func initDiskCaches(drives []string, cacheMaxUse int, t *testing.T) (*diskCache, error) {
var cfs []*cacheFSObjects
func initDiskCaches(drives []string, cacheMaxUse int, t *testing.T) ([]*diskCache, error) {
var cb []*diskCache
for _, d := range drives {
obj, err := initCacheFSObjects(d, cacheMaxUse)
obj, err := initCacheObjects(d, cacheMaxUse)
if err != nil {
return nil, err
}
cfs = append(cfs, obj)
cb = append(cb, obj)
}
return cb, nil
}
// Tests ToObjectInfo function.
func TestCacheMetadataObjInfo(t *testing.T) {
m := cacheMeta{Meta: nil}
objInfo := m.ToObjectInfo("testbucket", "testobject")
if objInfo.Size != 0 {
t.Fatal("Unexpected object info value for Size", objInfo.Size)
}
if objInfo.ModTime != timeSentinel {
t.Fatal("Unexpected object info value for ModTime ", objInfo.ModTime)
}
if objInfo.IsDir {
t.Fatal("Unexpected object info value for IsDir", objInfo.IsDir)
}
if !objInfo.Expires.IsZero() {
t.Fatal("Unexpected object info value for Expires ", objInfo.Expires)
}
return &diskCache{cfs: cfs}, nil
}
// test whether a drive being offline causes
// getCacheFS to fetch next online drive
func TestGetCacheFS(t *testing.T) {
// getCachedLoc to fetch next online drive
func TestGetCachedLoc(t *testing.T) {
for n := 1; n < 10; n++ {
fsDirs, err := getRandomDisks(n)
if err != nil {
@@ -56,14 +74,15 @@ func TestGetCacheFS(t *testing.T) {
if err != nil {
t.Fatal(err)
}
c := cacheObjects{cache: d}
bucketName := "testbucket"
objectName := "testobject"
ctx := context.Background()
// find cache drive where object would be hashed
index := d.hashIndex(bucketName, objectName)
index := c.hashIndex(bucketName, objectName)
// turn off drive by setting online status to false
d.cfs[index].online = false
cfs, err := d.getCacheFS(ctx, bucketName, objectName)
c.cache[index].online = false
cfs, err := c.getCacheLoc(ctx, bucketName, objectName)
if n == 1 && err == errDiskNotFound {
continue
}
@@ -71,7 +90,7 @@ func TestGetCacheFS(t *testing.T) {
t.Fatal(err)
}
i := -1
for j, f := range d.cfs {
for j, f := range c.cache {
if f == cfs {
i = j
break
@@ -84,8 +103,8 @@ func TestGetCacheFS(t *testing.T) {
}
// test whether a drive being offline causes
// getCacheFS to fetch next online drive
func TestGetCacheFSMaxUse(t *testing.T) {
// getCachedLoc to fetch next online drive
func TestGetCacheMaxUse(t *testing.T) {
for n := 1; n < 10; n++ {
fsDirs, err := getRandomDisks(n)
if err != nil {
@@ -95,14 +114,16 @@ func TestGetCacheFSMaxUse(t *testing.T) {
if err != nil {
t.Fatal(err)
}
c := cacheObjects{cache: d}
bucketName := "testbucket"
objectName := "testobject"
ctx := context.Background()
// find cache drive where object would be hashed
index := d.hashIndex(bucketName, objectName)
index := c.hashIndex(bucketName, objectName)
// turn off drive by setting online status to false
d.cfs[index].online = false
cfs, err := d.getCacheFS(ctx, bucketName, objectName)
c.cache[index].online = false
cb, err := c.getCacheLoc(ctx, bucketName, objectName)
if n == 1 && err == errDiskNotFound {
continue
}
@@ -110,8 +131,8 @@ func TestGetCacheFSMaxUse(t *testing.T) {
t.Fatal(err)
}
i := -1
for j, f := range d.cfs {
if f == cfs {
for j, f := range d {
if f == cb {
i = j
break
}
@@ -165,7 +186,9 @@ func TestDiskCache(t *testing.T) {
if err != nil {
t.Fatal(err)
}
cache := d.cfs[0]
c := cacheObjects{cache: d}
cache := c.cache[0]
ctx := context.Background()
bucketName := "testbucket"
objectName := "testobject"
@@ -191,14 +214,17 @@ func TestDiskCache(t *testing.T) {
if err != nil {
t.Fatal(err)
}
err = cache.Put(ctx, bucketName, objectName, NewPutObjReader(hashReader, nil, nil), ObjectOptions{UserDefined: httpMeta})
err = cache.Put(ctx, bucketName, objectName, hashReader, hashReader.Size(), ObjectOptions{UserDefined: httpMeta})
if err != nil {
t.Fatal(err)
}
cachedObjInfo, err := cache.GetObjectInfo(ctx, bucketName, objectName, opts)
cReader, err := cache.Get(ctx, bucketName, objectName, nil, http.Header{
"Content-Type": []string{"application/json"},
}, opts)
if err != nil {
t.Fatal(err)
}
cachedObjInfo := cReader.ObjInfo
if !cache.Exists(ctx, bucketName, objectName) {
t.Fatal("Expected object to exist on cache")
}
@@ -212,17 +238,16 @@ func TestDiskCache(t *testing.T) {
t.Fatal("Cached content-type does not match")
}
writer := bytes.NewBuffer(nil)
err = cache.Get(ctx, bucketName, objectName, 0, int64(size), writer, "", opts)
_, err = io.Copy(writer, cReader)
if err != nil {
t.Fatal(err)
}
if ccontent := writer.Bytes(); !bytes.Equal([]byte(content), ccontent) {
t.Errorf("wrong cached file content")
}
err = cache.Delete(ctx, bucketName, objectName)
if err != nil {
t.Errorf("object missing from cache")
}
cReader.Close()
cache.Delete(ctx, bucketName, objectName)
online := cache.IsOnline()
if !online {
t.Errorf("expected cache drive to be online")
@@ -239,7 +264,7 @@ func TestDiskCacheMaxUse(t *testing.T) {
if err != nil {
t.Fatal(err)
}
cache := d.cfs[0]
cache := d[0]
ctx := context.Background()
bucketName := "testbucket"
objectName := "testobject"
@@ -267,19 +292,20 @@ func TestDiskCacheMaxUse(t *testing.T) {
t.Fatal(err)
}
if !cache.diskAvailable(int64(size)) {
err = cache.Put(ctx, bucketName, objectName, NewPutObjReader(hashReader, nil, nil), ObjectOptions{UserDefined: httpMeta})
err = cache.Put(ctx, bucketName, objectName, hashReader, hashReader.Size(), ObjectOptions{UserDefined: httpMeta})
if err != errDiskFull {
t.Fatal("Cache max-use limit violated.")
}
} else {
err = cache.Put(ctx, bucketName, objectName, NewPutObjReader(hashReader, nil, nil), ObjectOptions{UserDefined: httpMeta})
err = cache.Put(ctx, bucketName, objectName, hashReader, hashReader.Size(), ObjectOptions{UserDefined: httpMeta})
if err != nil {
t.Fatal(err)
}
cachedObjInfo, err := cache.GetObjectInfo(ctx, bucketName, objectName, opts)
cReader, err := cache.Get(ctx, bucketName, objectName, nil, nil, opts)
if err != nil {
t.Fatal(err)
}
cachedObjInfo := cReader.ObjInfo
if !cache.Exists(ctx, bucketName, objectName) {
t.Fatal("Expected object to exist on cache")
}
@@ -293,92 +319,19 @@ func TestDiskCacheMaxUse(t *testing.T) {
t.Fatal("Cached content-type does not match")
}
writer := bytes.NewBuffer(nil)
err = cache.Get(ctx, bucketName, objectName, 0, int64(size), writer, "", opts)
_, err = io.Copy(writer, cReader)
if err != nil {
t.Fatal(err)
}
if ccontent := writer.Bytes(); !bytes.Equal([]byte(content), ccontent) {
t.Errorf("wrong cached file content")
}
err = cache.Delete(ctx, bucketName, objectName)
if err != nil {
t.Errorf("object missing from cache")
}
cReader.Close()
cache.Delete(ctx, bucketName, objectName)
online := cache.IsOnline()
if !online {
t.Errorf("expected cache drive to be online")
}
}
}
func TestIsCacheExcludeDirective(t *testing.T) {
testCases := []struct {
cacheControlOpt string
expectedResult bool
}{
{"no-cache", true},
{"no-store", true},
{"must-revalidate", true},
{"no-transform", false},
{"max-age", false},
}
for i, testCase := range testCases {
if isCacheExcludeDirective(testCase.cacheControlOpt) != testCase.expectedResult {
t.Errorf("Cache exclude directive test failed for case %d", i)
}
}
}
func TestGetCacheControlOpts(t *testing.T) {
testCases := []struct {
cacheControlHeaderVal string
expiryHeaderVal string
expectedCacheControl cacheControl
expectedErr bool
}{
{"", "", cacheControl{}, false},
{"max-age=2592000, public", "", cacheControl{maxAge: 2592000, sMaxAge: 0, minFresh: 0, expiry: time.Time{}, exclude: false}, false},
{"max-age=2592000, no-store", "", cacheControl{maxAge: 2592000, sMaxAge: 0, minFresh: 0, expiry: time.Time{}, exclude: true}, false},
{"must-revalidate, max-age=600", "", cacheControl{maxAge: 600, sMaxAge: 0, minFresh: 0, expiry: time.Time{}, exclude: true}, false},
{"s-maxAge=2500, max-age=600", "", cacheControl{maxAge: 600, sMaxAge: 2500, minFresh: 0, expiry: time.Time{}, exclude: false}, false},
{"s-maxAge=2500, max-age=600", "Wed, 21 Oct 2015 07:28:00 GMT", cacheControl{maxAge: 600, sMaxAge: 2500, minFresh: 0, expiry: time.Date(2015, time.October, 21, 07, 28, 00, 00, time.UTC), exclude: false}, false},
{"s-maxAge=2500, max-age=600s", "", cacheControl{maxAge: 600, sMaxAge: 2500, minFresh: 0, expiry: time.Time{}, exclude: false}, true},
}
var m map[string]string
for i, testCase := range testCases {
m = make(map[string]string)
m["cache-control"] = testCase.cacheControlHeaderVal
if testCase.expiryHeaderVal != "" {
m["expires"] = testCase.expiryHeaderVal
}
c, err := getCacheControlOpts(m)
if testCase.expectedErr && err == nil {
t.Errorf("expected err for case %d", i)
}
if !testCase.expectedErr && !reflect.DeepEqual(c, testCase.expectedCacheControl) {
t.Errorf("expected %v got %v for case %d", testCase.expectedCacheControl, c, i)
}
}
}
func TestFilterFromCache(t *testing.T) {
testCases := []struct {
metadata map[string]string
expectedResult bool
}{
{map[string]string{"content-type": "application/json"}, false},
{map[string]string{"cache-control": "private,no-store"}, true},
{map[string]string{"cache-control": "no-cache,must-revalidate"}, true},
{map[string]string{"cache-control": "no-transform"}, false},
{map[string]string{"cache-control": "max-age=3600"}, false},
}
for i, testCase := range testCases {
if filterFromCache(testCase.metadata) != testCase.expectedResult {
t.Errorf("Cache exclude directive test failed for case %d", i)
}
}
}
+2 -2
View File
@@ -23,7 +23,7 @@ import (
// getDiskUsage walks the file tree rooted at root, calling usageFn
// for each file or directory in the tree, including root.
func getDiskUsage(ctx context.Context, root string, usageFn usageFunc) error {
return walk(ctx, root+slashSeparator, usageFn)
return walk(ctx, root+SlashSeparator, usageFn)
}
type usageFunc func(ctx context.Context, entry string) error
@@ -34,7 +34,7 @@ func walk(ctx context.Context, path string, usageFn usageFunc) error {
return err
}
if !hasSuffix(path, slashSeparator) {
if !hasSuffix(path, SlashSeparator) {
return nil
}
+15 -6
View File
@@ -100,7 +100,7 @@ func (endpoint *Endpoint) UpdateIsLocal() error {
func NewEndpoint(arg string) (ep Endpoint, e error) {
// isEmptyPath - check whether given path is not empty.
isEmptyPath := func(path string) bool {
return path == "" || path == "/" || path == `\`
return path == "" || path == SlashSeparator || path == `\`
}
if isEmptyPath(arg) {
@@ -152,7 +152,7 @@ func NewEndpoint(arg string) (ep Endpoint, e error) {
return ep, fmt.Errorf("empty or root path is not supported in URL endpoint")
}
// On windows having a preceding "/" will cause problems, if the
// On windows having a preceding SlashSeparator will cause problems, if the
// command line already has C:/<export-folder/ in it. Final resulting
// path on windows might become C:/C:/ this will cause problems
// of starting minio server properly in distributed mode on windows.
@@ -720,18 +720,27 @@ func GetRemotePeers(endpoints EndpointList) []string {
func updateDomainIPs(endPoints set.StringSet) {
ipList := set.NewStringSet()
for e := range endPoints {
host, _, err := net.SplitHostPort(e)
host, port, err := net.SplitHostPort(e)
if err != nil {
if strings.Contains(err.Error(), "missing port in address") {
host = e
port = globalMinioPort
} else {
continue
}
}
IPs, _ := getHostIP(host)
ipList = ipList.Union(IPs)
IPs, err := getHostIP(host)
if err != nil {
continue
}
IPsWithPort := IPs.ApplyFunc(func(ip string) string {
return net.JoinHostPort(ip, port)
})
ipList = ipList.Union(IPsWithPort)
}
globalDomainIPs = ipList.FuncMatch(func(ip string, matchString string) bool {
return !strings.HasPrefix(ip, "127.") || strings.HasPrefix(ip, "::1")
return !(strings.HasPrefix(ip, "127.") || strings.HasPrefix(ip, "::1") || strings.HasPrefix(ip, "[::1]"))
}, "")
}
+42 -1
View File
@@ -25,6 +25,7 @@ import (
"testing"
"github.com/minio/cli"
"github.com/minio/minio-go/v6/pkg/set"
)
func TestSubOptimalEndpointInput(t *testing.T) {
@@ -95,7 +96,7 @@ func TestNewEndpoint(t *testing.T) {
{"http://127.0.0.1:8080/path", Endpoint{URL: u3, IsLocal: true, HostName: "127.0.0.1"}, URLEndpointType, nil},
{"http://192.168.253.200/path", Endpoint{URL: u4, IsLocal: false, HostName: "192.168.253.200"}, URLEndpointType, nil},
{"", Endpoint{}, -1, fmt.Errorf("empty or root endpoint is not supported")},
{"/", Endpoint{}, -1, fmt.Errorf("empty or root endpoint is not supported")},
{SlashSeparator, Endpoint{}, -1, fmt.Errorf("empty or root endpoint is not supported")},
{`\`, Endpoint{}, -1, fmt.Errorf("empty or root endpoint is not supported")},
{"c://foo", Endpoint{}, -1, fmt.Errorf("invalid URL endpoint format")},
{"ftp://foo", Endpoint{}, -1, fmt.Errorf("invalid URL endpoint format")},
@@ -444,3 +445,43 @@ func TestGetRemotePeers(t *testing.T) {
}
}
}
func TestUpdateDomainIPs(t *testing.T) {
tempGlobalMinioPort := globalMinioPort
defer func() {
globalMinioPort = tempGlobalMinioPort
}()
globalMinioPort = "9000"
tempGlobalDomainIPs := globalDomainIPs
defer func() {
globalDomainIPs = tempGlobalDomainIPs
}()
ipv4TestCases := []struct {
endPoints set.StringSet
expectedResult set.StringSet
}{
{set.NewStringSet(), set.NewStringSet()},
{set.CreateStringSet("localhost"), set.NewStringSet()},
{set.CreateStringSet("localhost", "10.0.0.1"), set.CreateStringSet("10.0.0.1:9000")},
{set.CreateStringSet("localhost:9001", "10.0.0.1"), set.CreateStringSet("10.0.0.1:9000")},
{set.CreateStringSet("localhost", "10.0.0.1:9001"), set.CreateStringSet("10.0.0.1:9001")},
{set.CreateStringSet("localhost:9000", "10.0.0.1:9001"), set.CreateStringSet("10.0.0.1:9001")},
{set.CreateStringSet("10.0.0.1", "10.0.0.2"), set.CreateStringSet("10.0.0.1:9000", "10.0.0.2:9000")},
{set.CreateStringSet("10.0.0.1:9001", "10.0.0.2"), set.CreateStringSet("10.0.0.1:9001", "10.0.0.2:9000")},
{set.CreateStringSet("10.0.0.1", "10.0.0.2:9002"), set.CreateStringSet("10.0.0.1:9000", "10.0.0.2:9002")},
{set.CreateStringSet("10.0.0.1:9001", "10.0.0.2:9002"), set.CreateStringSet("10.0.0.1:9001", "10.0.0.2:9002")},
}
for _, testCase := range ipv4TestCases {
globalDomainIPs = nil
updateDomainIPs(testCase.endPoints)
if !testCase.expectedResult.Equals(globalDomainIPs) {
t.Fatalf("error: expected = %s, got = %s", testCase.expectedResult, globalDomainIPs)
}
}
}
+72
View File
@@ -0,0 +1,72 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"errors"
"fmt"
etcd "github.com/coreos/etcd/clientv3"
)
var errEtcdUnreachable = errors.New("etcd is unreachable, please check your endpoints")
func etcdErrToErr(err error, etcdEndpoints []string) error {
if err == nil {
return nil
}
switch err {
case context.DeadlineExceeded:
return fmt.Errorf("%s %s", errEtcdUnreachable, etcdEndpoints)
default:
return fmt.Errorf("unexpected error %s from etcd, please check your endpoints %s", err, etcdEndpoints)
}
}
func saveKeyEtcd(ctx context.Context, client *etcd.Client, key string, data []byte) error {
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
_, err := client.Put(timeoutCtx, key, string(data))
return etcdErrToErr(err, client.Endpoints())
}
func deleteKeyEtcd(ctx context.Context, client *etcd.Client, key string) error {
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
_, err := client.Delete(timeoutCtx, key)
return etcdErrToErr(err, client.Endpoints())
}
func readKeyEtcd(ctx context.Context, client *etcd.Client, key string) ([]byte, error) {
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
resp, err := client.Get(timeoutCtx, key)
if err != nil {
return nil, etcdErrToErr(err, client.Endpoints())
}
if resp.Count == 0 {
return nil, errConfigNotFound
}
for _, ev := range resp.Kvs {
if string(ev.Key) == key {
return ev.Value, nil
}
}
return nil, errConfigNotFound
}
+210 -29
View File
@@ -18,11 +18,16 @@ package cmd
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"path"
"path/filepath"
"reflect"
"strings"
"github.com/minio/minio/cmd/logger"
)
@@ -33,6 +38,7 @@ const (
// formatCacheV1.Cache.Version
formatCacheVersionV1 = "1"
formatCacheVersionV2 = "2"
formatMetaVersion1 = "1"
@@ -56,6 +62,9 @@ type formatCacheV1 struct {
} `json:"cache"` // Cache field holds cache format.
}
// formatCacheV2 is same as formatCacheV1
type formatCacheV2 = formatCacheV1
// Used to detect the version of "cache" format.
type formatCacheVersionDetect struct {
Cache struct {
@@ -64,17 +73,17 @@ type formatCacheVersionDetect struct {
}
// Return a slice of format, to be used to format uninitialized disks.
func newFormatCacheV1(drives []string) []*formatCacheV1 {
func newFormatCacheV2(drives []string) []*formatCacheV2 {
diskCount := len(drives)
var disks = make([]string, diskCount)
var formats = make([]*formatCacheV1, diskCount)
var formats = make([]*formatCacheV2, diskCount)
for i := 0; i < diskCount; i++ {
format := &formatCacheV1{}
format := &formatCacheV2{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
format.Cache.Version = formatCacheVersionV2
format.Cache.DistributionAlgo = formatCacheV1DistributionAlgo
format.Cache.This = mustGetUUID()
formats[i] = format
@@ -87,6 +96,15 @@ func newFormatCacheV1(drives []string) []*formatCacheV1 {
return formats
}
// Returns formatCache.Cache.Version
func formatCacheGetVersion(r io.ReadSeeker) (string, error) {
format := &formatCacheVersionDetect{}
if err := jsonLoad(r, format); err != nil {
return "", err
}
return format.Cache.Version, nil
}
// Creates a new cache format.json if unformatted.
func createFormatCache(fsFormatPath string, format *formatCacheV1) error {
// open file using READ & WRITE permission
@@ -110,8 +128,8 @@ func createFormatCache(fsFormatPath string, format *formatCacheV1) error {
// This function creates a cache format file on disk and returns a slice
// of format cache config
func initFormatCache(ctx context.Context, drives []string) (formats []*formatCacheV1, err error) {
nformats := newFormatCacheV1(drives)
func initFormatCache(ctx context.Context, drives []string) (formats []*formatCacheV2, err error) {
nformats := newFormatCacheV2(drives)
for _, drive := range drives {
_, err = os.Stat(drive)
if err == nil {
@@ -147,26 +165,33 @@ func initFormatCache(ctx context.Context, drives []string) (formats []*formatCac
return nformats, nil
}
func loadFormatCache(ctx context.Context, drives []string) ([]*formatCacheV1, error) {
formats := make([]*formatCacheV1, len(drives))
func loadFormatCache(ctx context.Context, drives []string) ([]*formatCacheV2, bool, error) {
formats := make([]*formatCacheV2, len(drives))
var formatV2 *formatCacheV2
migrating := false
for i, drive := range drives {
cacheFormatPath := pathJoin(drive, minioMetaBucket, formatConfigFile)
f, err := os.Open(cacheFormatPath)
f, err := os.OpenFile(cacheFormatPath, os.O_RDWR, 0)
if err != nil {
if os.IsNotExist(err) {
continue
}
logger.LogIf(ctx, err)
return nil, err
return nil, migrating, err
}
defer f.Close()
format, err := formatMetaCacheV1(f)
if err != nil {
continue
}
formats[i] = format
formatV2 = format
if format.Cache.Version != formatCacheVersionV2 {
migrating = true
}
formats[i] = formatV2
}
return formats, nil
return formats, migrating, nil
}
// unmarshalls the cache format.json into formatCacheV1
@@ -178,26 +203,38 @@ func formatMetaCacheV1(r io.ReadSeeker) (*formatCacheV1, error) {
return format, nil
}
func checkFormatCacheValue(format *formatCacheV1) error {
func checkFormatCacheValue(format *formatCacheV2, migrating bool) error {
if format.Format != formatCache {
return fmt.Errorf("Unsupported cache format [%s] found", format.Format)
}
// during migration one or more cache drive(s) formats can be out of sync
if migrating {
// Validate format version and format type.
if format.Version != formatMetaVersion1 {
return fmt.Errorf("Unsupported version of cache format [%s] found", format.Version)
}
if format.Cache.Version != formatCacheVersionV2 && format.Cache.Version != formatCacheVersionV1 {
return fmt.Errorf("Unsupported Cache backend format found [%s]", format.Cache.Version)
}
return nil
}
// Validate format version and format type.
if format.Version != formatMetaVersion1 {
return fmt.Errorf("Unsupported version of cache format [%s] found", format.Version)
}
if format.Format != formatCache {
return fmt.Errorf("Unsupported cache format [%s] found", format.Format)
}
if format.Cache.Version != formatCacheVersionV1 {
if format.Cache.Version != formatCacheVersionV2 {
return fmt.Errorf("Unsupported Cache backend format found [%s]", format.Cache.Version)
}
return nil
}
func checkFormatCacheValues(formats []*formatCacheV1) (int, error) {
func checkFormatCacheValues(migrating bool, formats []*formatCacheV2) (int, error) {
for i, formatCache := range formats {
if formatCache == nil {
continue
}
if err := checkFormatCacheValue(formatCache); err != nil {
if err := checkFormatCacheValue(formatCache, migrating); err != nil {
return i, err
}
if len(formats) != len(formatCache.Cache.Disks) {
@@ -210,7 +247,7 @@ func checkFormatCacheValues(formats []*formatCacheV1) (int, error) {
// checkCacheDisksConsistency - checks if "This" disk uuid on each disk is consistent with all "Disks" slices
// across disks.
func checkCacheDiskConsistency(formats []*formatCacheV1) error {
func checkCacheDiskConsistency(formats []*formatCacheV2) error {
var disks = make([]string, len(formats))
// Collect currently available disk uuids.
for index, format := range formats {
@@ -236,7 +273,7 @@ func checkCacheDiskConsistency(formats []*formatCacheV1) error {
}
// checkCacheDisksSliceConsistency - validate cache Disks order if they are consistent.
func checkCacheDisksSliceConsistency(formats []*formatCacheV1) error {
func checkCacheDisksSliceConsistency(formats []*formatCacheV2) error {
var sentinelDisks []string
// Extract first valid Disks slice.
for _, format := range formats {
@@ -269,7 +306,7 @@ func findCacheDiskIndex(disk string, disks []string) int {
}
// validate whether cache drives order has changed
func validateCacheFormats(ctx context.Context, formats []*formatCacheV1) error {
func validateCacheFormats(ctx context.Context, migrating bool, formats []*formatCacheV2) error {
count := 0
for _, format := range formats {
if format == nil {
@@ -279,7 +316,7 @@ func validateCacheFormats(ctx context.Context, formats []*formatCacheV1) error {
if count == len(formats) {
return errors.New("Cache format files missing on all drives")
}
if _, err := checkFormatCacheValues(formats); err != nil {
if _, err := checkFormatCacheValues(migrating, formats); err != nil {
logger.LogIf(ctx, err)
return err
}
@@ -308,17 +345,161 @@ func cacheDrivesUnformatted(drives []string) bool {
// create format.json for each cache drive if fresh disk or load format from disk
// Then validate the format for all drives in the cache to ensure order
// of cache drives has not changed.
func loadAndValidateCacheFormat(ctx context.Context, drives []string) (formats []*formatCacheV1, err error) {
func loadAndValidateCacheFormat(ctx context.Context, drives []string) (formats []*formatCacheV2, migrating bool, err error) {
if cacheDrivesUnformatted(drives) {
formats, err = initFormatCache(ctx, drives)
} else {
formats, err = loadFormatCache(ctx, drives)
formats, migrating, err = loadFormatCache(ctx, drives)
}
if err != nil {
return nil, err
return nil, false, err
}
if err = validateCacheFormats(ctx, formats); err != nil {
return nil, err
if err = validateCacheFormats(ctx, migrating, formats); err != nil {
return nil, false, err
}
return formats, nil
return formats, migrating, nil
}
// reads cached object on disk and writes it back after adding bitrot
// hashsum per block as per the new disk cache format.
func migrateData(ctx context.Context, c *diskCache, oldfile, destDir string) error {
st, err := os.Stat(oldfile)
if err != nil {
err = osErrToFSFileErr(err)
return err
}
readCloser, err := readCacheFileStream(oldfile, 0, st.Size())
if err != nil {
return err
}
_, err = c.bitrotWriteToCache(ctx, destDir, readCloser, st.Size())
return err
}
// migrate cache contents from old cacheFS format to new backend format
// new format is flat
// sha(bucket,object)/ <== dir name
// - part.1 <== data
// - cache.json <== metadata
func migrateOldCache(ctx context.Context, c *diskCache) error {
oldCacheBucketsPath := path.Join(c.dir, minioMetaBucket, "buckets")
cacheFormatPath := pathJoin(c.dir, minioMetaBucket, formatConfigFile)
if _, err := os.Stat(oldCacheBucketsPath); err != nil {
// remove .minio.sys sub directories
removeAll(path.Join(c.dir, minioMetaBucket, "multipart"))
removeAll(path.Join(c.dir, minioMetaBucket, "tmp"))
removeAll(path.Join(c.dir, minioMetaBucket, "trash"))
removeAll(path.Join(c.dir, minioMetaBucket, "buckets"))
// just migrate cache format
return migrateCacheFormatJSON(cacheFormatPath)
}
buckets, err := readDir(oldCacheBucketsPath)
if err != nil {
return err
}
for _, bucket := range buckets {
var objMetaPaths []string
root := path.Join(oldCacheBucketsPath, bucket)
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if strings.HasSuffix(path, cacheMetaJSONFile) {
objMetaPaths = append(objMetaPaths, path)
}
return nil
})
if err != nil {
return err
}
for _, oMeta := range objMetaPaths {
objSlice := strings.SplitN(oMeta, cacheMetaJSONFile, 2)
object := strings.TrimPrefix(objSlice[0], path.Join(oldCacheBucketsPath, bucket))
object = strings.TrimSuffix(object, "/")
destdir := getCacheSHADir(c.dir, bucket, object)
if err := os.MkdirAll(destdir, 0777); err != nil {
return err
}
prevCachedPath := path.Join(c.dir, bucket, object)
// move cached object to new cache directory path
// migrate cache data and add bit-rot protection hash sum
// at the start of each block
if err := migrateData(ctx, c, prevCachedPath, destdir); err != nil {
continue
}
stat, err := os.Stat(prevCachedPath)
if err != nil {
if err == errFileNotFound {
continue
}
logger.LogIf(ctx, err)
return err
}
// old cached file can now be removed
if err := os.Remove(prevCachedPath); err != nil {
return err
}
// move cached metadata after changing cache metadata version
oldMetaPath := pathJoin(oldCacheBucketsPath, bucket, object, cacheMetaJSONFile)
metaPath := pathJoin(destdir, cacheMetaJSONFile)
metaBytes, err := ioutil.ReadFile(oldMetaPath)
if err != nil {
return err
}
// marshal cache metadata after adding version and stat info
meta := &cacheMeta{}
if err = json.Unmarshal(metaBytes, &meta); err != nil {
return err
}
meta.Checksum = CacheChecksumInfoV1{Algorithm: HighwayHash256S.String(), Blocksize: cacheBlkSize}
meta.Version = cacheMetaVersion
meta.Stat.Size = stat.Size()
meta.Stat.ModTime = stat.ModTime()
jsonData, err := json.Marshal(meta)
if err != nil {
return err
}
if err = ioutil.WriteFile(metaPath, jsonData, 0644); err != nil {
return err
}
}
// delete old bucket from cache, now that all contents are cleared
removeAll(path.Join(c.dir, bucket))
}
// remove .minio.sys sub directories
removeAll(path.Join(c.dir, minioMetaBucket, "multipart"))
removeAll(path.Join(c.dir, minioMetaBucket, "tmp"))
removeAll(path.Join(c.dir, minioMetaBucket, "trash"))
removeAll(path.Join(c.dir, minioMetaBucket, "buckets"))
return migrateCacheFormatJSON(cacheFormatPath)
}
func migrateCacheFormatJSON(cacheFormatPath string) error {
// now migrate format.json
f, err := os.OpenFile(cacheFormatPath, os.O_RDWR, 0)
if err != nil {
return err
}
defer f.Close()
formatV1 := formatCacheV1{}
if err := jsonLoad(f, &formatV1); err != nil {
return err
}
formatV2 := &formatCacheV2{}
formatV2.formatMetaV1 = formatV1.formatMetaV1
formatV2.Version = formatMetaVersion1
formatV2.Cache = formatV1.Cache
formatV2.Cache.Version = formatCacheVersionV2
if err := jsonSave(f, formatV2); err != nil {
return err
}
return nil
}
+27 -37
View File
@@ -18,20 +18,10 @@ package cmd
import (
"context"
"io"
"os"
"testing"
)
// Returns format.Cache.Version
func formatCacheGetVersion(r io.ReadSeeker) (string, error) {
format := &formatCacheVersionDetect{}
if err := jsonLoad(r, format); err != nil {
return "", err
}
return format.Cache.Version, nil
}
// TestDiskCacheFormat - tests initFormatCache, formatMetaGetFormatBackendCache, formatCacheGetVersion.
func TestDiskCacheFormat(t *testing.T) {
ctx := context.Background()
@@ -55,8 +45,8 @@ func TestDiskCacheFormat(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if version != formatCacheVersionV1 {
t.Fatalf(`expected: %s, got: %s`, formatCacheVersionV1, version)
if version != formatCacheVersionV2 {
t.Fatalf(`expected: %s, got: %s`, formatCacheVersionV2, version)
}
// Corrupt the format.json file and test the functions.
@@ -68,7 +58,7 @@ func TestDiskCacheFormat(t *testing.T) {
t.Fatal(err)
}
if _, err = loadAndValidateCacheFormat(context.Background(), fsDirs); err == nil {
if _, _, err = loadAndValidateCacheFormat(context.Background(), fsDirs); err == nil {
t.Fatal("expected to fail")
}
@@ -81,15 +71,15 @@ func TestDiskCacheFormat(t *testing.T) {
t.Fatal(err)
}
if _, err = loadAndValidateCacheFormat(context.Background(), fsDirs); err == nil {
if _, _, err = loadAndValidateCacheFormat(context.Background(), fsDirs); err == nil {
t.Fatal("expected to fail")
}
}
// generates a valid format.json for Cache backend.
func genFormatCacheValid() []*formatCacheV1 {
func genFormatCacheValid() []*formatCacheV2 {
disks := make([]string, 8)
formatConfigs := make([]*formatCacheV1, 8)
formatConfigs := make([]*formatCacheV2, 8)
for index := range disks {
disks[index] = mustGetUUID()
}
@@ -97,7 +87,7 @@ func genFormatCacheValid() []*formatCacheV1 {
format := &formatCacheV1{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
format.Cache.Version = formatCacheVersionV2
format.Cache.This = disks[index]
format.Cache.Disks = disks
formatConfigs[index] = format
@@ -106,9 +96,9 @@ func genFormatCacheValid() []*formatCacheV1 {
}
// generates a invalid format.json version for Cache backend.
func genFormatCacheInvalidVersion() []*formatCacheV1 {
func genFormatCacheInvalidVersion() []*formatCacheV2 {
disks := make([]string, 8)
formatConfigs := make([]*formatCacheV1, 8)
formatConfigs := make([]*formatCacheV2, 8)
for index := range disks {
disks[index] = mustGetUUID()
}
@@ -128,14 +118,14 @@ func genFormatCacheInvalidVersion() []*formatCacheV1 {
}
// generates a invalid format.json version for Cache backend.
func genFormatCacheInvalidFormat() []*formatCacheV1 {
func genFormatCacheInvalidFormat() []*formatCacheV2 {
disks := make([]string, 8)
formatConfigs := make([]*formatCacheV1, 8)
formatConfigs := make([]*formatCacheV2, 8)
for index := range disks {
disks[index] = mustGetUUID()
}
for index := range disks {
format := &formatCacheV1{}
format := &formatCacheV2{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
@@ -150,14 +140,14 @@ func genFormatCacheInvalidFormat() []*formatCacheV1 {
}
// generates a invalid format.json version for Cache backend.
func genFormatCacheInvalidCacheVersion() []*formatCacheV1 {
func genFormatCacheInvalidCacheVersion() []*formatCacheV2 {
disks := make([]string, 8)
formatConfigs := make([]*formatCacheV1, 8)
formatConfigs := make([]*formatCacheV2, 8)
for index := range disks {
disks[index] = mustGetUUID()
}
for index := range disks {
format := &formatCacheV1{}
format := &formatCacheV2{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
@@ -172,17 +162,17 @@ func genFormatCacheInvalidCacheVersion() []*formatCacheV1 {
}
// generates a invalid format.json version for Cache backend.
func genFormatCacheInvalidDisksCount() []*formatCacheV1 {
func genFormatCacheInvalidDisksCount() []*formatCacheV2 {
disks := make([]string, 7)
formatConfigs := make([]*formatCacheV1, 8)
formatConfigs := make([]*formatCacheV2, 8)
for index := range disks {
disks[index] = mustGetUUID()
}
for index := range disks {
format := &formatCacheV1{}
format := &formatCacheV2{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
format.Cache.Version = formatCacheVersionV2
format.Cache.This = disks[index]
format.Cache.Disks = disks
formatConfigs[index] = format
@@ -191,9 +181,9 @@ func genFormatCacheInvalidDisksCount() []*formatCacheV1 {
}
// generates a invalid format.json Disks for Cache backend.
func genFormatCacheInvalidDisks() []*formatCacheV1 {
func genFormatCacheInvalidDisks() []*formatCacheV2 {
disks := make([]string, 8)
formatConfigs := make([]*formatCacheV1, 8)
formatConfigs := make([]*formatCacheV2, 8)
for index := range disks {
disks[index] = mustGetUUID()
}
@@ -201,7 +191,7 @@ func genFormatCacheInvalidDisks() []*formatCacheV1 {
format := &formatCacheV1{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
format.Cache.Version = formatCacheVersionV2
format.Cache.This = disks[index]
format.Cache.Disks = disks
formatConfigs[index] = format
@@ -226,7 +216,7 @@ func genFormatCacheInvalidThis() []*formatCacheV1 {
format := &formatCacheV1{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
format.Cache.Version = formatCacheVersionV2
format.Cache.This = disks[index]
format.Cache.Disks = disks
formatConfigs[index] = format
@@ -238,9 +228,9 @@ func genFormatCacheInvalidThis() []*formatCacheV1 {
}
// generates a invalid format.json Disk UUID in wrong order for Cache backend.
func genFormatCacheInvalidDisksOrder() []*formatCacheV1 {
func genFormatCacheInvalidDisksOrder() []*formatCacheV2 {
disks := make([]string, 8)
formatConfigs := make([]*formatCacheV1, 8)
formatConfigs := make([]*formatCacheV2, 8)
for index := range disks {
disks[index] = mustGetUUID()
}
@@ -248,7 +238,7 @@ func genFormatCacheInvalidDisksOrder() []*formatCacheV1 {
format := &formatCacheV1{}
format.Version = formatMetaVersion1
format.Format = formatCache
format.Cache.Version = formatCacheVersionV1
format.Cache.Version = formatCacheVersionV2
format.Cache.This = disks[index]
format.Cache.Disks = disks
formatConfigs[index] = format
@@ -319,7 +309,7 @@ func TestFormatCache(t *testing.T) {
}
for i, testCase := range testCases {
err := validateCacheFormats(context.Background(), testCase.formatConfigs)
err := validateCacheFormats(context.Background(), false, testCase.formatConfigs)
if err != nil && testCase.shouldPass {
t.Errorf("Test %d: Expected to pass but failed with %s", i+1, err)
}
+16
View File
@@ -698,6 +698,22 @@ func initStorageDisks(endpoints EndpointList) ([]StorageAPI, error) {
return storageDisks, nil
}
// Runs through the faulty disks and record their errors.
func initDisksWithErrors(endpoints EndpointList) ([]StorageAPI, []error) {
storageDisks := make([]StorageAPI, len(endpoints))
var dErrs = make([]error, len(storageDisks))
for index, endpoint := range endpoints {
storage, err := newStorageAPI(endpoint)
if err != nil {
logger.LogIf(context.Background(), err)
dErrs[index] = err
continue
}
storageDisks[index] = storage
}
return storageDisks, dErrs
}
// formatXLV3ThisEmpty - find out if '.This' field is empty
// in any of the input `formats`, if yes return true.
func formatXLV3ThisEmpty(formats []*formatXLV3) bool {
-14
View File
@@ -228,15 +228,6 @@ func fsStatDir(ctx context.Context, statDir string) (os.FileInfo, error) {
return fi, nil
}
// Returns if the dirPath is a directory.
func fsIsDir(ctx context.Context, dirPath string) bool {
fi, err := fsStat(ctx, dirPath)
if err != nil {
return false
}
return fi.IsDir()
}
// Lookup if file exists, returns file attributes upon success.
func fsStatFile(ctx context.Context, statFile string) (os.FileInfo, error) {
fi, err := fsStat(ctx, statFile)
@@ -418,11 +409,6 @@ func fsRenameFile(ctx context.Context, sourcePath, destPath string) error {
return err
}
// Verify if source path exists.
if _, err := os.Stat(sourcePath); err != nil {
return osErrToFSFileErr(err)
}
if err := renameAll(sourcePath, destPath); err != nil {
logger.LogIf(ctx, err)
return err
-12
View File
@@ -548,18 +548,6 @@ func TestFSRemoveMeta(t *testing.T) {
}
}
func TestFSIsDir(t *testing.T) {
dirPath, err := ioutil.TempDir(globalTestTmpDir, "minio-")
if err != nil {
t.Fatalf("Unable to create tmp directory %s", err)
}
defer os.RemoveAll(dirPath)
if !fsIsDir(context.Background(), dirPath) {
t.Fatalf("Expected %s to be a directory", dirPath)
}
}
func TestFSIsFile(t *testing.T) {
dirPath, err := ioutil.TempDir(globalTestTmpDir, "minio-")
if err != nil {
+1 -1
View File
@@ -141,7 +141,7 @@ func (m fsMetaV1) ToObjectInfo(bucket, object string, fi os.FileInfo) ObjectInfo
m.Meta["content-type"] = mimedb.TypeByExtension(pathutil.Ext(object))
}
if hasSuffix(object, slashSeparator) {
if hasSuffix(object, SlashSeparator) {
m.Meta["etag"] = emptyETag // For directories etag is d41d8cd98f00b204e9800998ecf8427e
m.Meta["content-type"] = "application/octet-stream"
}
+1 -1
View File
@@ -163,7 +163,7 @@ func (fs *FSObjects) ListMultipartUploads(ctx context.Context, bucket, object, k
}
uploads = append(uploads, MultipartInfo{
Object: object,
UploadID: strings.TrimSuffix(uploadID, slashSeparator),
UploadID: strings.TrimSuffix(uploadID, SlashSeparator),
Initiated: fi.ModTime(),
})
}
+5 -9
View File
@@ -505,7 +505,7 @@ func (fs *FSObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
return nil, toObjectErr(err, bucket, object)
}
// For a directory, we need to send an reader that returns no bytes.
if hasSuffix(object, slashSeparator) {
if hasSuffix(object, SlashSeparator) {
// The lock taken above is released when
// objReader.Close() is called by the caller.
return NewGetObjectReaderFromReader(bytes.NewBuffer(nil), objInfo, opts.CheckCopyPrecondFn, nsUnlocker)
@@ -596,7 +596,7 @@ func (fs *FSObjects) getObject(ctx context.Context, bucket, object string, offse
}
// If its a directory request, we return an empty body.
if hasSuffix(object, slashSeparator) {
if hasSuffix(object, SlashSeparator) {
_, err = writer.Write([]byte(""))
logger.LogIf(ctx, err)
return toObjectErr(err, bucket, object)
@@ -690,11 +690,7 @@ func (fs *FSObjects) defaultFsJSON(object string) fsMetaV1 {
// getObjectInfo - wrapper for reading object metadata and constructs ObjectInfo.
func (fs *FSObjects) getObjectInfo(ctx context.Context, bucket, object string) (oi ObjectInfo, e error) {
fsMeta := fsMetaV1{}
if hasSuffix(object, slashSeparator) {
// Since we support PUT of a "directory" object, we allow HEAD.
if !fsIsDir(ctx, pathJoin(fs.fsPath, bucket, object)) {
return oi, errFileNotFound
}
if hasSuffix(object, SlashSeparator) {
fi, err := fsStatDir(ctx, pathJoin(fs.fsPath, bucket, object))
if err != nil {
return oi, err
@@ -754,7 +750,7 @@ func (fs *FSObjects) getObjectInfoWithLock(ctx context.Context, bucket, object s
return oi, err
}
if strings.HasSuffix(object, slashSeparator) && !fs.isObjectDir(bucket, object) {
if strings.HasSuffix(object, SlashSeparator) && !fs.isObjectDir(bucket, object) {
return oi, errFileNotFound
}
@@ -788,7 +784,7 @@ func (fs *FSObjects) GetObjectInfo(ctx context.Context, bucket, object string, o
func (fs *FSObjects) parentDirIsObject(ctx context.Context, bucket, parent string) bool {
var isParentDirObject func(string) bool
isParentDirObject = func(p string) bool {
if p == "." || p == "/" {
if p == "." || p == SlashSeparator {
return false
}
if fsIsFile(ctx, pathJoin(fs.fsPath, bucket, p)) {
+2 -2
View File
@@ -71,7 +71,7 @@ func TestFSParentDirIsObject(t *testing.T) {
// Should not cause infinite loop.
{
parentIsObject: false,
objectName: "/",
objectName: SlashSeparator,
},
{
parentIsObject: false,
@@ -214,7 +214,7 @@ func TestFSPutObject(t *testing.T) {
}
// With a directory object.
_, err = obj.PutObject(context.Background(), bucketName+"non-existent", objectName+"/", mustGetPutObjReader(t, bytes.NewReader([]byte("abcd")), 0, "", ""), ObjectOptions{})
_, err = obj.PutObject(context.Background(), bucketName+"non-existent", objectName+SlashSeparator, mustGetPutObjReader(t, bytes.NewReader([]byte("abcd")), 0, "", ""), ObjectOptions{})
if err == nil {
t.Fatal("Unexpected should fail here, bucket doesn't exist")
}
+12 -16
View File
@@ -106,6 +106,7 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
logger.Disable = true
// Validate if we have access, secret set through environment.
globalGatewayName = gw.Name()
gatewayName := gw.Name()
if ctx.Args().First() == "help" {
cli.ShowCommandHelpAndExit(ctx, gatewayName, 1)
@@ -158,7 +159,7 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
registerSTSRouter(router)
}
enableConfigOps := globalEtcdClient != nil && gatewayName == "nas"
enableConfigOps := gatewayName == "nas"
enableIAMOps := globalEtcdClient != nil
// Enable IAM admin APIs if etcd is enabled, if not just enable basic
@@ -236,6 +237,10 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// Load globalServerConfig from etcd
logger.LogIf(context.Background(), globalConfigSys.Init(newObject))
// Start watching disk for reloading config, this
// is only enabled for "NAS" gateway.
globalConfigSys.WatchConfigNASDisk(newObject)
}
// Load logger subsystem
@@ -249,7 +254,7 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
if len(cacheConfig.Drives) > 0 {
var err error
// initialize the new disk cache objects.
globalCacheObjectAPI, err = newServerCacheObjects(cacheConfig)
globalCacheObjectAPI, err = newServerCacheObjects(context.Background(), cacheConfig)
logger.FatalIf(err, "Unable to initialize disk caching")
}
@@ -274,23 +279,14 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// Create new notification system.
globalNotificationSys = NewNotificationSys(globalServerConfig, globalEndpoints)
if globalEtcdClient != nil && newObject.IsNotificationSupported() {
if enableConfigOps && newObject.IsNotificationSupported() {
logger.LogIf(context.Background(), globalNotificationSys.Init(newObject))
}
// Encryption support checks in gateway mode.
{
if (globalAutoEncryption || GlobalKMS != nil) && !newObject.IsEncryptionSupported() {
logger.Fatal(errInvalidArgument,
"Encryption support is requested but (%s) gateway does not support encryption", gw.Name())
}
if GlobalGatewaySSE.IsSet() && GlobalKMS == nil {
logger.Fatal(uiErrInvalidGWSSEEnvValue(nil).Msg("MINIO_GATEWAY_SSE set but KMS is not configured"),
"Unable to start gateway with SSE")
}
}
// Verify if object layer supports
// - encryption
// - compression
verifyObjectLayerFeatures("gateway "+gatewayName, newObject)
// Once endpoints are finalized, initialize the new object api.
globalObjLayerMutex.Lock()
+1 -1
View File
@@ -431,7 +431,7 @@ func checkAzureUploadID(ctx context.Context, uploadID string) (err error) {
// parses partID from part metadata file name
func parseAzurePart(metaPartFileName, prefix string) (partID int, err error) {
partStr := strings.TrimPrefix(metaPartFileName, prefix+"/")
partStr := strings.TrimPrefix(metaPartFileName, prefix+minio.SlashSeparator)
if partID, err = strconv.Atoi(partStr); err != nil || partID <= 0 {
err = fmt.Errorf("invalid part number in block id '%s'", string(partID))
return
+3 -3
View File
@@ -472,7 +472,7 @@ func (l *gcsGateway) ListBuckets(ctx context.Context) (buckets []minio.BucketInf
// DeleteBucket delete a bucket on GCS.
func (l *gcsGateway) DeleteBucket(ctx context.Context, bucket string) error {
itObject := l.client.Bucket(bucket).Objects(ctx, &storage.Query{
Delimiter: "/",
Delimiter: minio.SlashSeparator,
Versions: false,
})
// We list the bucket and if we find any objects we return BucketNotEmpty error. If we
@@ -1040,7 +1040,7 @@ func (l *gcsGateway) ListMultipartUploads(ctx context.Context, bucket string, pr
if prefix == mpMeta.Object {
// Extract uploadId
// E.g minio.sys.tmp/multipart/v1/d063ad89-fdc4-4ea3-a99e-22dba98151f5/gcs.json
components := strings.SplitN(attrs.Name, "/", 5)
components := strings.SplitN(attrs.Name, minio.SlashSeparator, 5)
if len(components) != 5 {
compErr := errors.New("Invalid multipart upload format")
logger.LogIf(ctx, compErr)
@@ -1114,7 +1114,7 @@ func (l *gcsGateway) PutObjectPart(ctx context.Context, bucket string, key strin
// gcsGetPartInfo returns PartInfo of a given object part
func gcsGetPartInfo(ctx context.Context, attrs *storage.ObjectAttrs) (minio.PartInfo, error) {
components := strings.SplitN(attrs.Name, "/", 5)
components := strings.SplitN(attrs.Name, minio.SlashSeparator, 5)
if len(components) != 5 {
logger.LogIf(ctx, errors.New("Invalid multipart upload format"))
return minio.PartInfo{}, errors.New("Invalid multipart upload format")
+1 -1
View File
@@ -36,7 +36,7 @@ const (
// Ignores all reserved bucket names or invalid bucket names.
func isReservedOrInvalidBucket(bucketEntry string, strict bool) bool {
bucketEntry = strings.TrimSuffix(bucketEntry, "/")
bucketEntry = strings.TrimSuffix(bucketEntry, minio.SlashSeparator)
if strict {
if err := s3utils.CheckValidBucketNameStrict(bucketEntry); err != nil {
return true
+76 -17
View File
@@ -18,6 +18,7 @@ package hdfs
import (
"context"
"fmt"
"io"
"net"
"net/http"
@@ -30,7 +31,11 @@ import (
"time"
"github.com/minio/cli"
krb "github.com/minio/gokrb5/v7/client"
"github.com/minio/gokrb5/v7/config"
"github.com/minio/gokrb5/v7/credentials"
"github.com/minio/hdfs/v3"
"github.com/minio/hdfs/v3/hadoopconf"
"github.com/minio/minio-go/v6/pkg/s3utils"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/cmd/logger"
@@ -41,7 +46,7 @@ import (
const (
hdfsBackend = "hdfs"
hdfsSeparator = "/"
hdfsSeparator = minio.SlashSeparator
)
func init() {
@@ -102,7 +107,7 @@ EXAMPLES:
// Handler for 'minio gateway hdfs' command line.
func hdfsGatewayMain(ctx *cli.Context) {
// Validate gateway arguments.
if !ctx.Args().Present() || ctx.Args().First() == "help" {
if ctx.Args().First() == "help" {
cli.ShowCommandHelpAndExit(ctx, hdfsBackend, 1)
}
@@ -119,6 +124,43 @@ func (g *HDFS) Name() string {
return hdfsBackend
}
func getKerberosClient() (*krb.Client, error) {
configPath := os.Getenv("KRB5_CONFIG")
if configPath == "" {
configPath = "/etc/krb5.conf"
}
cfg, err := config.Load(configPath)
if err != nil {
return nil, err
}
// Determine the ccache location from the environment,
// falling back to the default location.
ccachePath := os.Getenv("KRB5CCNAME")
if strings.Contains(ccachePath, ":") {
if strings.HasPrefix(ccachePath, "FILE:") {
ccachePath = strings.TrimPrefix(ccachePath, "FILE:")
} else {
return nil, fmt.Errorf("unable to use kerberos ccache: %s", ccachePath)
}
} else if ccachePath == "" {
u, err := user.Current()
if err != nil {
return nil, err
}
ccachePath = fmt.Sprintf("/tmp/krb5cc_%s", u.Uid)
}
ccache, err := credentials.LoadCCache(ccachePath)
if err != nil {
return nil, err
}
return krb.NewClientFromCCache(ccache, cfg)
}
// NewGatewayLayer returns hdfs gatewaylayer.
func (g *HDFS) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error) {
dialFunc := (&net.Dialer{
@@ -127,25 +169,42 @@ func (g *HDFS) NewGatewayLayer(creds auth.Credentials) (minio.ObjectLayer, error
DualStack: true,
}).DialContext
var addresses []string
for _, s := range g.args {
u, err := xnet.ParseURL(s)
if err != nil {
return nil, err
}
addresses = append(addresses, u.Host)
}
user, err := user.Current()
hconfig, err := hadoopconf.LoadFromEnvironment()
if err != nil {
return nil, err
}
opts := hdfs.ClientOptions{
Addresses: addresses,
User: user.Username,
NamenodeDialFunc: dialFunc,
DatanodeDialFunc: dialFunc,
opts := hdfs.ClientOptionsFromConf(hconfig)
opts.NamenodeDialFunc = dialFunc
opts.DatanodeDialFunc = dialFunc
// Not addresses found, load it from command line.
if len(opts.Addresses) == 0 {
var addresses []string
for _, s := range g.args {
u, err := xnet.ParseURL(s)
if err != nil {
return nil, err
}
addresses = append(addresses, u.Host)
}
opts.Addresses = addresses
}
if opts.KerberosClient != nil {
opts.KerberosClient, err = getKerberosClient()
if err != nil {
return nil, fmt.Errorf("Unable to initialize kerberos client: %s", err)
}
} else {
opts.User = os.Getenv("HADOOP_USER_NAME")
if opts.User == "" {
u, err := user.Current()
if err != nil {
return nil, fmt.Errorf("Unable to lookup local user: %s", err)
}
opts.User = u.Username
}
}
clnt, err := hdfs.NewClient(opts)
+6 -7
View File
@@ -41,7 +41,6 @@ const (
// custom multipart files are stored under the defaultMinioGWPrefix
defaultMinioGWPrefix = ".minio"
defaultGWContentFileName = "data"
slashSeparator = "/"
)
// s3EncObjects is a wrapper around s3Objects and implements gateway calls for
@@ -102,7 +101,7 @@ func (l *s3EncObjects) ListObjectsV2(ctx context.Context, bucket, prefix, contin
}
// get objectname and ObjectInfo from the custom metadata file
if strings.HasSuffix(obj.Name, gwdareMetaJSON) {
objSlice := strings.Split(obj.Name, slashSeparator+defaultMinioGWPrefix)
objSlice := strings.Split(obj.Name, minio.SlashSeparator+defaultMinioGWPrefix)
gwMeta, e := l.getGWMetadata(ctx, bucket, getDareMetaPath(objSlice[0]))
if e != nil {
continue
@@ -117,7 +116,7 @@ func (l *s3EncObjects) ListObjectsV2(ctx context.Context, bucket, prefix, contin
}
}
for _, p := range loi.Prefixes {
objName := strings.TrimSuffix(p, slashSeparator)
objName := strings.TrimSuffix(p, minio.SlashSeparator)
gm, err := l.getGWMetadata(ctx, bucket, getDareMetaPath(objName))
// if prefix is actually a custom multi-part object, append it to objects
if err == nil {
@@ -165,7 +164,7 @@ func isGWObject(objName string) bool {
return false
}
pfxSlice := strings.Split(objName, slashSeparator)
pfxSlice := strings.Split(objName, minio.SlashSeparator)
var i1, i2 int
for i := len(pfxSlice) - 1; i >= 0; i-- {
p := pfxSlice[i]
@@ -401,10 +400,10 @@ func (l *s3EncObjects) ListMultipartUploads(ctx context.Context, bucket string,
if e != nil {
return
}
lmi.KeyMarker = strings.TrimSuffix(lmi.KeyMarker, getGWContentPath("/"))
lmi.NextKeyMarker = strings.TrimSuffix(lmi.NextKeyMarker, getGWContentPath("/"))
lmi.KeyMarker = strings.TrimSuffix(lmi.KeyMarker, getGWContentPath(minio.SlashSeparator))
lmi.NextKeyMarker = strings.TrimSuffix(lmi.NextKeyMarker, getGWContentPath(minio.SlashSeparator))
for i := range lmi.Uploads {
lmi.Uploads[i].Object = strings.TrimSuffix(lmi.Uploads[i].Object, getGWContentPath("/"))
lmi.Uploads[i].Object = strings.TrimSuffix(lmi.Uploads[i].Object, getGWContentPath(minio.SlashSeparator))
}
return
}
+13 -8
View File
@@ -153,7 +153,7 @@ func containsReservedMetadata(header http.Header) bool {
// Reserved bucket.
const (
minioReservedBucket = "minio"
minioReservedBucketPath = "/" + minioReservedBucket
minioReservedBucketPath = SlashSeparator + minioReservedBucket
)
// Adds redirect rules for incoming requests.
@@ -172,13 +172,15 @@ func setBrowserRedirectHandler(h http.Handler) http.Handler {
// browser requests.
func getRedirectLocation(urlPath string) (rLocation string) {
if urlPath == minioReservedBucketPath {
rLocation = minioReservedBucketPath + "/"
rLocation = minioReservedBucketPath + SlashSeparator
}
if contains([]string{
"/",
SlashSeparator,
"/webrpc",
"/login",
"/favicon.ico",
"/favicon-16x16.png",
"/favicon-32x32.png",
"/favicon-96x96.png",
}, urlPath) {
rLocation = minioReservedBucketPath + urlPath
}
@@ -229,7 +231,7 @@ func guessIsRPCReq(req *http.Request) bool {
return false
}
return req.Method == http.MethodPost &&
strings.HasPrefix(req.URL.Path, minioReservedBucketPath+"/")
strings.HasPrefix(req.URL.Path, minioReservedBucketPath+SlashSeparator)
}
func (h redirectHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
@@ -258,7 +260,7 @@ func setBrowserCacheControlHandler(h http.Handler) http.Handler {
func (h cacheControlHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet && guessIsBrowserReq(r) {
// For all browser requests set appropriate Cache-Control policies
if hasPrefix(r.URL.Path, minioReservedBucketPath+"/") {
if hasPrefix(r.URL.Path, minioReservedBucketPath+SlashSeparator) {
if hasSuffix(r.URL.Path, ".js") || r.URL.Path == minioReservedBucketPath+"/favicon.ico" {
// For assets set cache expiry of one year. For each release, the name
// of the asset name will change and hence it can not be served from cache.
@@ -276,7 +278,7 @@ func (h cacheControlHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Check to allow access to the reserved "bucket" `/minio` for Admin
// API requests.
func isAdminReq(r *http.Request) bool {
return strings.HasPrefix(r.URL.Path, adminAPIPathPrefix+"/")
return strings.HasPrefix(r.URL.Path, adminAPIPathPrefix+SlashSeparator)
}
// Adds verification for incoming paths.
@@ -596,7 +598,7 @@ const (
// such as ".." and "."
func hasBadPathComponent(path string) bool {
path = strings.TrimSpace(path)
for _, p := range strings.Split(path, slashSeparator) {
for _, p := range strings.Split(path, SlashSeparator) {
switch strings.TrimSpace(p) {
case dotdotComponent:
return true
@@ -746,6 +748,9 @@ func setBucketForwardingHandler(h http.Handler) http.Handler {
fwd := handlers.NewForwarder(&handlers.Forwarder{
PassHost: true,
RoundTripper: NewCustomHTTPTransport(),
Logger: func(err error) {
logger.LogIf(context.Background(), err)
},
})
return bucketForwardingHandler{fwd, h}
}
+17 -7
View File
@@ -35,12 +35,12 @@ func TestRedirectLocation(t *testing.T) {
{
// 1. When urlPath is '/minio'
urlPath: minioReservedBucketPath,
location: minioReservedBucketPath + "/",
location: minioReservedBucketPath + SlashSeparator,
},
{
// 2. When urlPath is '/'
urlPath: "/",
location: minioReservedBucketPath + "/",
urlPath: SlashSeparator,
location: minioReservedBucketPath + SlashSeparator,
},
{
// 3. When urlPath is '/webrpc'
@@ -53,12 +53,22 @@ func TestRedirectLocation(t *testing.T) {
location: minioReservedBucketPath + "/login",
},
{
// 5. When urlPath is '/favicon.ico'
urlPath: "/favicon.ico",
location: minioReservedBucketPath + "/favicon.ico",
// 5. When urlPath is '/favicon-16x16.png'
urlPath: "/favicon-16x16.png",
location: minioReservedBucketPath + "/favicon-16x16.png",
},
{
// 6. When urlPath is '/unknown'
// 6. When urlPath is '/favicon-16x16.png'
urlPath: "/favicon-32x32.png",
location: minioReservedBucketPath + "/favicon-32x32.png",
},
{
// 7. When urlPath is '/favicon-96x96.png'
urlPath: "/favicon-96x96.png",
location: minioReservedBucketPath + "/favicon-96x96.png",
},
{
// 8. When urlPath is '/unknown'
urlPath: "/unknown",
location: "",
},
+3
View File
@@ -118,6 +118,9 @@ var (
// Indicates if the running minio is in gateway mode.
globalIsGateway = false
// Name of gateway server, e.g S3, GCS, Azure, etc
globalGatewayName = ""
// This flag is set to 'true' by default
globalIsBrowserEnabled = true
+3 -20
View File
@@ -95,8 +95,8 @@ func isMetadataReplace(h http.Header) bool {
// Splits an incoming path into bucket and object components.
func path2BucketAndObject(path string) (bucket, object string) {
// Skip the first element if it is '/', split the rest.
path = strings.TrimPrefix(path, "/")
pathComponents := strings.SplitN(path, "/", 2)
path = strings.TrimPrefix(path, SlashSeparator)
pathComponents := strings.SplitN(path, SlashSeparator, 2)
// Save the bucket and object extracted from path.
switch len(pathComponents) {
@@ -370,7 +370,7 @@ func getResource(path string, host string, domains []string) (string, error) {
continue
}
bucket := strings.TrimSuffix(host, "."+domain)
return slashSeparator + pathJoin(bucket, path), nil
return SlashSeparator + pathJoin(bucket, path), nil
}
return path, nil
}
@@ -394,20 +394,3 @@ func getHostName(r *http.Request) (hostName string) {
}
return
}
func isHTTPStatusOK(statusCode int) bool {
// List of success status.
var successStatus = []int{
http.StatusOK,
http.StatusCreated,
http.StatusAccepted,
http.StatusNoContent,
http.StatusPartialContent,
}
for _, okstatus := range successStatus {
if statusCode == okstatus {
return true
}
}
return false
}
+10 -8
View File
@@ -26,14 +26,16 @@ import (
)
const (
minioHealthGoroutineThreshold = 1000
minioHealthGoroutineThreshold = 10000
)
// ReadinessCheckHandler -- checks if there are more than threshold number of goroutines running,
// returns service unavailable.
// Readiness probes are used to detect situations where application is under heavy load
// and temporarily unable to serve. In a orchestrated setup like Kubernetes, containers reporting
// that they are not ready do not receive traffic through Kubernetes Services.
// ReadinessCheckHandler -- checks if there are more than threshold
// number of goroutines running, returns service unavailable.
//
// Readiness probes are used to detect situations where application
// is under heavy load and temporarily unable to serve. In a orchestrated
// setup like Kubernetes, containers reporting that they are not ready do
// not receive traffic through Kubernetes Services.
func ReadinessCheckHandler(w http.ResponseWriter, r *http.Request) {
if err := goroutineCountCheck(minioHealthGoroutineThreshold); err != nil {
writeResponse(w, http.StatusServiceUnavailable, nil, mimeNone)
@@ -98,8 +100,8 @@ func LivenessCheckHandler(w http.ResponseWriter, r *http.Request) {
writeResponse(w, http.StatusOK, nil, mimeNone)
}
// checks threshold against total number of go-routines in the system and throws error if
// more than threshold go-routines are running.
// checks threshold against total number of go-routines in the system and
// throws error if more than threshold go-routines are running.
func goroutineCountCheck(threshold int) error {
count := runtime.NumGoroutine()
if count > threshold {
+610
View File
@@ -0,0 +1,610 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"encoding/json"
"errors"
"path"
"strings"
"sync"
"time"
etcd "github.com/coreos/etcd/clientv3"
"github.com/coreos/etcd/mvcc/mvccpb"
"github.com/minio/minio-go/v6/pkg/set"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
iampolicy "github.com/minio/minio/pkg/iam/policy"
)
var defaultContextTimeout = 30 * time.Second
func etcdKvsToSet(prefix string, kvs []*mvccpb.KeyValue) set.StringSet {
users := set.NewStringSet()
for _, kv := range kvs {
// Extract user by stripping off the `prefix` value as suffix,
// then strip off the remaining basename to obtain the prefix
// value, usually in the following form.
//
// key := "config/iam/users/newuser/identity.json"
// prefix := "config/iam/users/"
// v := trim(trim(key, prefix), base(key)) == "newuser"
//
user := path.Clean(strings.TrimSuffix(strings.TrimPrefix(string(kv.Key), prefix), path.Base(string(kv.Key))))
users.Add(user)
}
return users
}
func etcdKvsToSetPolicyDB(prefix string, kvs []*mvccpb.KeyValue) set.StringSet {
items := set.NewStringSet()
for _, kv := range kvs {
// Extract user item by stripping off prefix and then
// stripping of ".json" suffix.
//
// key := "config/iam/policydb/users/myuser1.json"
// prefix := "config/iam/policydb/users/"
// v := trimSuffix(trimPrefix(key, prefix), ".json")
key := string(kv.Key)
item := path.Clean(strings.TrimSuffix(strings.TrimPrefix(key, prefix), ".json"))
items.Add(item)
}
return items
}
// IAMEtcdStore implements IAMStorageAPI
type IAMEtcdStore struct {
sync.RWMutex
ctx context.Context
client *etcd.Client
}
func newIAMEtcdStore() *IAMEtcdStore {
return &IAMEtcdStore{client: globalEtcdClient}
}
func (ies *IAMEtcdStore) getContext() context.Context {
ies.RLock()
defer ies.RUnlock()
if ies.ctx == nil {
return context.Background()
}
return ies.ctx
}
func (ies *IAMEtcdStore) setContext(ctx context.Context) {
ies.Lock()
defer ies.Unlock()
ies.ctx = ctx
}
func (ies *IAMEtcdStore) clearContext() {
ies.Lock()
defer ies.Unlock()
ies.ctx = nil
}
func (ies *IAMEtcdStore) saveIAMConfig(item interface{}, path string) error {
data, err := json.Marshal(item)
if err != nil {
return err
}
return saveKeyEtcd(ies.getContext(), ies.client, path, data)
}
func (ies *IAMEtcdStore) loadIAMConfig(item interface{}, path string) error {
pdata, err := readKeyEtcd(ies.getContext(), ies.client, path)
if err != nil {
return err
}
return json.Unmarshal(pdata, item)
}
func (ies *IAMEtcdStore) deleteIAMConfig(path string) error {
return deleteKeyEtcd(ies.getContext(), ies.client, path)
}
func (ies *IAMEtcdStore) migrateUsersConfigToV1(isSTS bool) error {
basePrefix := iamConfigUsersPrefix
if isSTS {
basePrefix = iamConfigSTSPrefix
}
ctx, cancel := context.WithTimeout(context.Background(), defaultContextTimeout)
defer cancel()
ies.setContext(ctx)
defer ies.clearContext()
r, err := ies.client.Get(ctx, basePrefix, etcd.WithPrefix(), etcd.WithKeysOnly())
if err != nil {
return err
}
users := etcdKvsToSet(basePrefix, r.Kvs)
for _, user := range users.ToSlice() {
{
// 1. check if there is a policy file in the old loc.
oldPolicyPath := pathJoin(basePrefix, user, iamPolicyFile)
var policyName string
err := ies.loadIAMConfig(&policyName, oldPolicyPath)
if err != nil {
switch err {
case errConfigNotFound:
// No mapped policy or already migrated.
default:
// corrupt data/read error, etc
}
goto next
}
// 2. copy policy to new loc.
mp := newMappedPolicy(policyName)
path := getMappedPolicyPath(user, isSTS, false)
if err := ies.saveIAMConfig(mp, path); err != nil {
return err
}
// 3. delete policy file in old loc.
deleteKeyEtcd(ctx, ies.client, oldPolicyPath)
}
next:
// 4. check if user identity has old format.
identityPath := pathJoin(basePrefix, user, iamIdentityFile)
var cred auth.Credentials
if err := ies.loadIAMConfig(&cred, identityPath); err != nil {
switch err {
case errConfigNotFound:
// This case should not happen.
default:
// corrupt file or read error
}
continue
}
// If the file is already in the new format,
// then the parsed auth.Credentials will have
// the zero value for the struct.
var zeroCred auth.Credentials
if cred == zeroCred {
// nothing to do
continue
}
// Found a id file in old format. Copy value
// into new format and save it.
cred.AccessKey = user
u := newUserIdentity(cred)
if err := ies.saveIAMConfig(u, identityPath); err != nil {
logger.LogIf(context.Background(), err)
return err
}
// Nothing to delete as identity file location
// has not changed.
}
return nil
}
func (ies *IAMEtcdStore) migrateToV1() error {
var iamFmt iamFormat
path := getIAMFormatFilePath()
if err := ies.loadIAMConfig(&iamFmt, path); err != nil {
switch err {
case errConfigNotFound:
// Need to migrate to V1.
default:
return errors.New("corrupt IAM format file")
}
} else {
if iamFmt.Version >= iamFormatVersion1 {
// Already migrated to V1 of higher!
return nil
}
// This case should not happen
// (i.e. Version is 0 or negative.)
return errors.New("got an invalid IAM format version")
}
// Migrate long-term users
if err := ies.migrateUsersConfigToV1(false); err != nil {
logger.LogIf(context.Background(), err)
return err
}
// Migrate STS users
if err := ies.migrateUsersConfigToV1(true); err != nil {
logger.LogIf(context.Background(), err)
return err
}
// Save iam version file.
if err := ies.saveIAMConfig(newIAMFormatVersion1(), path); err != nil {
logger.LogIf(context.Background(), err)
return err
}
return nil
}
// Should be called under config migration lock
func (ies *IAMEtcdStore) migrateBackendFormat(objAPI ObjectLayer) error {
if err := ies.migrateToV1(); err != nil {
return err
}
return nil
}
func (ies *IAMEtcdStore) loadPolicyDoc(policy string, m map[string]iampolicy.Policy) error {
var p iampolicy.Policy
err := ies.loadIAMConfig(&p, getPolicyDocPath(policy))
if err != nil {
return err
}
m[policy] = p
return nil
}
func (ies *IAMEtcdStore) loadPolicyDocs(m map[string]iampolicy.Policy) error {
ctx, cancel := context.WithTimeout(context.Background(), defaultContextTimeout)
defer cancel()
ies.setContext(ctx)
defer ies.clearContext()
r, err := ies.client.Get(ctx, iamConfigPoliciesPrefix, etcd.WithPrefix(), etcd.WithKeysOnly())
if err != nil {
return err
}
policies := etcdKvsToSet(iamConfigPoliciesPrefix, r.Kvs)
// Reload config and policies for all policys.
for _, policyName := range policies.ToSlice() {
err = ies.loadPolicyDoc(policyName, m)
if err != nil {
return err
}
}
return nil
}
func (ies *IAMEtcdStore) loadUser(user string, isSTS bool, m map[string]auth.Credentials) error {
var u UserIdentity
err := ies.loadIAMConfig(&u, getUserIdentityPath(user, isSTS))
if err != nil {
return err
}
if u.Credentials.IsExpired() {
// Delete expired identity.
ctx := ies.getContext()
deleteKeyEtcd(ctx, ies.client, getUserIdentityPath(user, isSTS))
deleteKeyEtcd(ctx, ies.client, getMappedPolicyPath(user, isSTS, false))
return nil
}
if u.Credentials.AccessKey == "" {
u.Credentials.AccessKey = user
}
m[user] = u.Credentials
return nil
}
func (ies *IAMEtcdStore) loadUsers(isSTS bool, m map[string]auth.Credentials) error {
basePrefix := iamConfigUsersPrefix
if isSTS {
basePrefix = iamConfigSTSPrefix
}
ctx, cancel := context.WithTimeout(context.Background(), defaultContextTimeout)
defer cancel()
ies.setContext(ctx)
defer ies.clearContext()
r, err := ies.client.Get(ctx, basePrefix, etcd.WithPrefix(), etcd.WithKeysOnly())
if err != nil {
return err
}
users := etcdKvsToSet(basePrefix, r.Kvs)
// Reload config for all users.
for _, user := range users.ToSlice() {
if err = ies.loadUser(user, isSTS, m); err != nil {
return err
}
}
return nil
}
func (ies *IAMEtcdStore) loadGroup(group string, m map[string]GroupInfo) error {
var gi GroupInfo
err := ies.loadIAMConfig(&gi, getGroupInfoPath(group))
if err != nil {
return err
}
m[group] = gi
return nil
}
func (ies *IAMEtcdStore) loadGroups(m map[string]GroupInfo) error {
ctx, cancel := context.WithTimeout(context.Background(), defaultContextTimeout)
defer cancel()
ies.setContext(ctx)
defer ies.clearContext()
r, err := ies.client.Get(ctx, iamConfigGroupsPrefix, etcd.WithPrefix(), etcd.WithKeysOnly())
if err != nil {
return err
}
groups := etcdKvsToSet(iamConfigGroupsPrefix, r.Kvs)
// Reload config for all groups.
for _, group := range groups.ToSlice() {
if err = ies.loadGroup(group, m); err != nil {
return err
}
}
return nil
}
func (ies *IAMEtcdStore) loadMappedPolicy(name string, isSTS, isGroup bool, m map[string]MappedPolicy) error {
var p MappedPolicy
err := ies.loadIAMConfig(&p, getMappedPolicyPath(name, isSTS, isGroup))
if err != nil {
return err
}
m[name] = p
return nil
}
func (ies *IAMEtcdStore) loadMappedPolicies(isSTS, isGroup bool, m map[string]MappedPolicy) error {
ctx, cancel := context.WithTimeout(context.Background(), defaultContextTimeout)
defer cancel()
ies.setContext(ctx)
defer ies.clearContext()
var basePrefix string
switch {
case isSTS:
basePrefix = iamConfigPolicyDBSTSUsersPrefix
case isGroup:
basePrefix = iamConfigPolicyDBGroupsPrefix
default:
basePrefix = iamConfigPolicyDBUsersPrefix
}
r, err := ies.client.Get(ctx, basePrefix, etcd.WithPrefix(), etcd.WithKeysOnly())
if err != nil {
return err
}
users := etcdKvsToSetPolicyDB(basePrefix, r.Kvs)
// Reload config and policies for all users.
for _, user := range users.ToSlice() {
if err = ies.loadMappedPolicy(user, isSTS, isGroup, m); err != nil {
return err
}
}
return nil
}
func (ies *IAMEtcdStore) loadAll(sys *IAMSys, objectAPI ObjectLayer) error {
iamUsersMap := make(map[string]auth.Credentials)
iamGroupsMap := make(map[string]GroupInfo)
iamPolicyDocsMap := make(map[string]iampolicy.Policy)
iamUserPolicyMap := make(map[string]MappedPolicy)
iamGroupPolicyMap := make(map[string]MappedPolicy)
if err := ies.loadPolicyDocs(iamPolicyDocsMap); err != nil {
return err
}
if err := ies.loadUsers(false, iamUsersMap); err != nil {
return err
}
// load STS temp users into the same map
if err := ies.loadUsers(true, iamUsersMap); err != nil {
return err
}
if err := ies.loadGroups(iamGroupsMap); err != nil {
return err
}
if err := ies.loadMappedPolicies(false, false, iamUserPolicyMap); err != nil {
return err
}
// load STS policy mappings into the same map
if err := ies.loadMappedPolicies(true, false, iamUserPolicyMap); err != nil {
return err
}
// load policies mapped to groups
if err := ies.loadMappedPolicies(false, true, iamGroupPolicyMap); err != nil {
return err
}
// Sets default canned policies, if none are set.
setDefaultCannedPolicies(iamPolicyDocsMap)
sys.Lock()
defer sys.Unlock()
sys.iamUsersMap = iamUsersMap
sys.iamGroupsMap = iamGroupsMap
sys.iamUserPolicyMap = iamUserPolicyMap
sys.iamPolicyDocsMap = iamPolicyDocsMap
sys.iamGroupPolicyMap = iamGroupPolicyMap
sys.buildUserGroupMemberships()
return nil
}
func (ies *IAMEtcdStore) savePolicyDoc(policyName string, p iampolicy.Policy) error {
return ies.saveIAMConfig(&p, getPolicyDocPath(policyName))
}
func (ies *IAMEtcdStore) saveMappedPolicy(name string, isSTS, isGroup bool, mp MappedPolicy) error {
return ies.saveIAMConfig(mp, getMappedPolicyPath(name, isSTS, isGroup))
}
func (ies *IAMEtcdStore) saveUserIdentity(name string, isSTS bool, u UserIdentity) error {
return ies.saveIAMConfig(u, getUserIdentityPath(name, isSTS))
}
func (ies *IAMEtcdStore) saveGroupInfo(name string, gi GroupInfo) error {
return ies.saveIAMConfig(gi, getGroupInfoPath(name))
}
func (ies *IAMEtcdStore) deletePolicyDoc(name string) error {
return ies.deleteIAMConfig(getPolicyDocPath(name))
}
func (ies *IAMEtcdStore) deleteMappedPolicy(name string, isSTS, isGroup bool) error {
return ies.deleteIAMConfig(getMappedPolicyPath(name, isSTS, isGroup))
}
func (ies *IAMEtcdStore) deleteUserIdentity(name string, isSTS bool) error {
return ies.deleteIAMConfig(getUserIdentityPath(name, isSTS))
}
func (ies *IAMEtcdStore) deleteGroupInfo(name string) error {
return ies.deleteIAMConfig(getGroupInfoPath(name))
}
func (ies *IAMEtcdStore) watch(sys *IAMSys) {
watchEtcd := func() {
// Refresh IAMSys with etcd watch.
for {
watchCh := ies.client.Watch(context.Background(),
iamConfigPrefix, etcd.WithPrefix(), etcd.WithKeysOnly())
select {
case <-GlobalServiceDoneCh:
return
case watchResp, ok := <-watchCh:
if !ok {
time.Sleep(1 * time.Second)
continue
}
if err := watchResp.Err(); err != nil {
logger.LogIf(context.Background(), err)
// log and retry.
time.Sleep(1 * time.Second)
continue
}
for _, event := range watchResp.Events {
sys.Lock()
ies.reloadFromEvent(sys, event)
sys.Unlock()
}
}
}
}
go watchEtcd()
}
// sys.RLock is held by caller.
func (ies *IAMEtcdStore) reloadFromEvent(sys *IAMSys, event *etcd.Event) {
eventCreate := event.IsModify() || event.IsCreate()
eventDelete := event.Type == etcd.EventTypeDelete
usersPrefix := strings.HasPrefix(string(event.Kv.Key), iamConfigUsersPrefix)
groupsPrefix := strings.HasPrefix(string(event.Kv.Key), iamConfigGroupsPrefix)
stsPrefix := strings.HasPrefix(string(event.Kv.Key), iamConfigSTSPrefix)
policyPrefix := strings.HasPrefix(string(event.Kv.Key), iamConfigPoliciesPrefix)
policyDBUsersPrefix := strings.HasPrefix(string(event.Kv.Key), iamConfigPolicyDBUsersPrefix)
policyDBSTSUsersPrefix := strings.HasPrefix(string(event.Kv.Key), iamConfigPolicyDBSTSUsersPrefix)
policyDBGroupsPrefix := strings.HasPrefix(string(event.Kv.Key), iamConfigPolicyDBGroupsPrefix)
switch {
case eventCreate:
switch {
case usersPrefix:
accessKey := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigUsersPrefix))
ies.loadUser(accessKey, false, sys.iamUsersMap)
case stsPrefix:
accessKey := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigSTSPrefix))
ies.loadUser(accessKey, true, sys.iamUsersMap)
case groupsPrefix:
group := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigGroupsPrefix))
ies.loadGroup(group, sys.iamGroupsMap)
gi := sys.iamGroupsMap[group]
sys.removeGroupFromMembershipsMap(group)
sys.updateGroupMembershipsMap(group, &gi)
case policyPrefix:
policyName := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigPoliciesPrefix))
ies.loadPolicyDoc(policyName, sys.iamPolicyDocsMap)
case policyDBUsersPrefix:
policyMapFile := strings.TrimPrefix(string(event.Kv.Key),
iamConfigPolicyDBUsersPrefix)
user := strings.TrimSuffix(policyMapFile, ".json")
ies.loadMappedPolicy(user, false, false, sys.iamUserPolicyMap)
case policyDBSTSUsersPrefix:
policyMapFile := strings.TrimPrefix(string(event.Kv.Key),
iamConfigPolicyDBSTSUsersPrefix)
user := strings.TrimSuffix(policyMapFile, ".json")
ies.loadMappedPolicy(user, true, false, sys.iamUserPolicyMap)
case policyDBGroupsPrefix:
policyMapFile := strings.TrimPrefix(string(event.Kv.Key),
iamConfigPolicyDBGroupsPrefix)
user := strings.TrimSuffix(policyMapFile, ".json")
ies.loadMappedPolicy(user, false, true, sys.iamGroupPolicyMap)
}
case eventDelete:
switch {
case usersPrefix:
accessKey := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigUsersPrefix))
delete(sys.iamUsersMap, accessKey)
case stsPrefix:
accessKey := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigSTSPrefix))
delete(sys.iamUsersMap, accessKey)
case groupsPrefix:
group := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigGroupsPrefix))
sys.removeGroupFromMembershipsMap(group)
delete(sys.iamGroupsMap, group)
delete(sys.iamGroupPolicyMap, group)
case policyPrefix:
policyName := path.Dir(strings.TrimPrefix(string(event.Kv.Key),
iamConfigPoliciesPrefix))
delete(sys.iamPolicyDocsMap, policyName)
case policyDBUsersPrefix:
policyMapFile := strings.TrimPrefix(string(event.Kv.Key),
iamConfigPolicyDBUsersPrefix)
user := strings.TrimSuffix(policyMapFile, ".json")
delete(sys.iamUserPolicyMap, user)
case policyDBSTSUsersPrefix:
policyMapFile := strings.TrimPrefix(string(event.Kv.Key),
iamConfigPolicyDBSTSUsersPrefix)
user := strings.TrimSuffix(policyMapFile, ".json")
delete(sys.iamUserPolicyMap, user)
case policyDBGroupsPrefix:
policyMapFile := strings.TrimPrefix(string(event.Kv.Key),
iamConfigPolicyDBGroupsPrefix)
user := strings.TrimSuffix(policyMapFile, ".json")
delete(sys.iamGroupPolicyMap, user)
}
}
}
+582
View File
@@ -0,0 +1,582 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
iampolicy "github.com/minio/minio/pkg/iam/policy"
)
// IAMObjectStore implements IAMStorageAPI
type IAMObjectStore struct {
// Protect assignment to objAPI
sync.RWMutex
objAPI ObjectLayer
}
func newIAMObjectStore() *IAMObjectStore {
return &IAMObjectStore{objAPI: nil}
}
func (iamOS *IAMObjectStore) getObjectAPI() ObjectLayer {
iamOS.RLock()
defer iamOS.RUnlock()
if iamOS.objAPI != nil {
return iamOS.objAPI
}
return newObjectLayerFn()
}
func (iamOS *IAMObjectStore) setObjectAPI(objAPI ObjectLayer) {
iamOS.Lock()
defer iamOS.Unlock()
iamOS.objAPI = objAPI
}
func (iamOS *IAMObjectStore) clearObjectAPI() {
iamOS.Lock()
defer iamOS.Unlock()
iamOS.objAPI = nil
}
// Migrate users directory in a single scan.
//
// 1. Migrate user policy from:
//
// `iamConfigUsersPrefix + "<username>/policy.json"`
//
// to:
//
// `iamConfigPolicyDBUsersPrefix + "<username>.json"`.
//
// 2. Add versioning to the policy json file in the new
// location.
//
// 3. Migrate user identity json file to include version info.
func (iamOS *IAMObjectStore) migrateUsersConfigToV1(isSTS bool) error {
basePrefix := iamConfigUsersPrefix
if isSTS {
basePrefix = iamConfigSTSPrefix
}
objAPI := iamOS.getObjectAPI()
doneCh := make(chan struct{})
defer close(doneCh)
for item := range listIAMConfigItems(objAPI, basePrefix, true, doneCh) {
if item.Err != nil {
return item.Err
}
user := item.Item
{
// 1. check if there is policy file in old location.
oldPolicyPath := pathJoin(basePrefix, user, iamPolicyFile)
var policyName string
if err := iamOS.loadIAMConfig(&policyName, oldPolicyPath); err != nil {
switch err {
case errConfigNotFound:
// This case means it is already
// migrated or there is no policy on
// user.
default:
// File may be corrupt or network error
}
// Nothing to do on the policy file,
// so move on to check the id file.
goto next
}
// 2. copy policy file to new location.
mp := newMappedPolicy(policyName)
if err := iamOS.saveMappedPolicy(user, isSTS, false, mp); err != nil {
return err
}
// 3. delete policy file from old
// location. Ignore error.
iamOS.deleteIAMConfig(oldPolicyPath)
}
next:
// 4. check if user identity has old format.
identityPath := pathJoin(basePrefix, user, iamIdentityFile)
var cred auth.Credentials
if err := iamOS.loadIAMConfig(&cred, identityPath); err != nil {
switch err.(type) {
case ObjectNotFound:
// This should not happen.
default:
// File may be corrupt or network error
}
continue
}
// If the file is already in the new format,
// then the parsed auth.Credentials will have
// the zero value for the struct.
var zeroCred auth.Credentials
if cred == zeroCred {
// nothing to do
continue
}
// Found a id file in old format. Copy value
// into new format and save it.
cred.AccessKey = user
u := newUserIdentity(cred)
if err := iamOS.saveIAMConfig(u, identityPath); err != nil {
logger.LogIf(context.Background(), err)
return err
}
// Nothing to delete as identity file location
// has not changed.
}
return nil
}
func (iamOS *IAMObjectStore) migrateToV1() error {
var iamFmt iamFormat
path := getIAMFormatFilePath()
if err := iamOS.loadIAMConfig(&iamFmt, path); err != nil {
switch err {
case errConfigNotFound:
// Need to migrate to V1.
default:
return errors.New("corrupt IAM format file")
}
} else {
if iamFmt.Version >= iamFormatVersion1 {
// Nothing to do.
return nil
}
// This case should not happen
// (i.e. Version is 0 or negative.)
return errors.New("got an invalid IAM format version")
}
// Migrate long-term users
if err := iamOS.migrateUsersConfigToV1(false); err != nil {
logger.LogIf(context.Background(), err)
return err
}
// Migrate STS users
if err := iamOS.migrateUsersConfigToV1(true); err != nil {
logger.LogIf(context.Background(), err)
return err
}
// Save iam format to version 1.
if err := iamOS.saveIAMConfig(newIAMFormatVersion1(), path); err != nil {
logger.LogIf(context.Background(), err)
return err
}
return nil
}
// Should be called under config migration lock
func (iamOS *IAMObjectStore) migrateBackendFormat(objAPI ObjectLayer) error {
iamOS.setObjectAPI(objAPI)
defer iamOS.clearObjectAPI()
if err := iamOS.migrateToV1(); err != nil {
return err
}
return nil
}
func (iamOS *IAMObjectStore) saveIAMConfig(item interface{}, path string) error {
objectAPI := iamOS.getObjectAPI()
data, err := json.Marshal(item)
if err != nil {
return err
}
return saveConfig(context.Background(), objectAPI, path, data)
}
func (iamOS *IAMObjectStore) loadIAMConfig(item interface{}, path string) error {
objectAPI := iamOS.getObjectAPI()
data, err := readConfig(context.Background(), objectAPI, path)
if err != nil {
return err
}
return json.Unmarshal(data, item)
}
func (iamOS *IAMObjectStore) deleteIAMConfig(path string) error {
err := deleteConfig(context.Background(), iamOS.getObjectAPI(), path)
if _, ok := err.(ObjectNotFound); ok {
return errConfigNotFound
}
return err
}
func (iamOS *IAMObjectStore) loadPolicyDoc(policy string, m map[string]iampolicy.Policy) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
var p iampolicy.Policy
err := iamOS.loadIAMConfig(&p, getPolicyDocPath(policy))
if err != nil {
return err
}
m[policy] = p
return nil
}
func (iamOS *IAMObjectStore) loadPolicyDocs(m map[string]iampolicy.Policy) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
doneCh := make(chan struct{})
defer close(doneCh)
for item := range listIAMConfigItems(objectAPI, iamConfigPoliciesPrefix, true, doneCh) {
if item.Err != nil {
return item.Err
}
policyName := item.Item
err := iamOS.loadPolicyDoc(policyName, m)
if err != nil {
return err
}
}
return nil
}
func (iamOS *IAMObjectStore) loadUser(user string, isSTS bool, m map[string]auth.Credentials) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
var u UserIdentity
err := iamOS.loadIAMConfig(&u, getUserIdentityPath(user, isSTS))
if err != nil {
return err
}
if u.Credentials.IsExpired() {
// Delete expired identity - ignoring errors here.
iamOS.deleteIAMConfig(getUserIdentityPath(user, isSTS))
iamOS.deleteIAMConfig(getMappedPolicyPath(user, isSTS, false))
return nil
}
if u.Credentials.AccessKey == "" {
u.Credentials.AccessKey = user
}
m[user] = u.Credentials
return nil
}
func (iamOS *IAMObjectStore) loadUsers(isSTS bool, m map[string]auth.Credentials) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
doneCh := make(chan struct{})
defer close(doneCh)
basePrefix := iamConfigUsersPrefix
if isSTS {
basePrefix = iamConfigSTSPrefix
}
for item := range listIAMConfigItems(objectAPI, basePrefix, true, doneCh) {
if item.Err != nil {
return item.Err
}
userName := item.Item
err := iamOS.loadUser(userName, isSTS, m)
if err != nil {
return err
}
}
return nil
}
func (iamOS *IAMObjectStore) loadGroup(group string, m map[string]GroupInfo) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
var g GroupInfo
err := iamOS.loadIAMConfig(&g, getGroupInfoPath(group))
if err != nil {
return err
}
m[group] = g
return nil
}
func (iamOS *IAMObjectStore) loadGroups(m map[string]GroupInfo) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
doneCh := make(chan struct{})
defer close(doneCh)
for item := range listIAMConfigItems(objectAPI, iamConfigGroupsPrefix, true, doneCh) {
if item.Err != nil {
return item.Err
}
group := item.Item
err := iamOS.loadGroup(group, m)
if err != nil {
return err
}
}
return nil
}
func (iamOS *IAMObjectStore) loadMappedPolicy(name string, isSTS, isGroup bool,
m map[string]MappedPolicy) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
var p MappedPolicy
err := iamOS.loadIAMConfig(&p, getMappedPolicyPath(name, isSTS, isGroup))
if err != nil {
return err
}
m[name] = p
return nil
}
func (iamOS *IAMObjectStore) loadMappedPolicies(isSTS, isGroup bool, m map[string]MappedPolicy) error {
objectAPI := iamOS.getObjectAPI()
if objectAPI == nil {
return errServerNotInitialized
}
doneCh := make(chan struct{})
defer close(doneCh)
var basePath string
switch {
case isSTS:
basePath = iamConfigPolicyDBSTSUsersPrefix
case isGroup:
basePath = iamConfigPolicyDBGroupsPrefix
default:
basePath = iamConfigPolicyDBUsersPrefix
}
for item := range listIAMConfigItems(objectAPI, basePath, false, doneCh) {
if item.Err != nil {
return item.Err
}
policyFile := item.Item
userOrGroupName := strings.TrimSuffix(policyFile, ".json")
err := iamOS.loadMappedPolicy(userOrGroupName, isSTS, isGroup, m)
if err != nil {
return err
}
}
return nil
}
// Refresh IAMSys. If an object layer is passed in use that, otherwise
// load from global.
func (iamOS *IAMObjectStore) loadAll(sys *IAMSys, objectAPI ObjectLayer) error {
if objectAPI == nil {
objectAPI = iamOS.getObjectAPI()
}
if objectAPI == nil {
return errServerNotInitialized
}
// cache object layer for other load* functions
iamOS.setObjectAPI(objectAPI)
defer iamOS.clearObjectAPI()
iamUsersMap := make(map[string]auth.Credentials)
iamGroupsMap := make(map[string]GroupInfo)
iamPolicyDocsMap := make(map[string]iampolicy.Policy)
iamUserPolicyMap := make(map[string]MappedPolicy)
iamGroupPolicyMap := make(map[string]MappedPolicy)
if err := iamOS.loadPolicyDocs(iamPolicyDocsMap); err != nil {
return err
}
if err := iamOS.loadUsers(false, iamUsersMap); err != nil {
return err
}
// load STS temp users into the same map
if err := iamOS.loadUsers(true, iamUsersMap); err != nil {
return err
}
if err := iamOS.loadGroups(iamGroupsMap); err != nil {
return err
}
if err := iamOS.loadMappedPolicies(false, false, iamUserPolicyMap); err != nil {
return err
}
// load STS policy mappings into the same map
if err := iamOS.loadMappedPolicies(true, false, iamUserPolicyMap); err != nil {
return err
}
// load policies mapped to groups
if err := iamOS.loadMappedPolicies(false, true, iamGroupPolicyMap); err != nil {
return err
}
// Sets default canned policies, if none are set.
setDefaultCannedPolicies(iamPolicyDocsMap)
sys.Lock()
defer sys.Unlock()
sys.iamUsersMap = iamUsersMap
sys.iamPolicyDocsMap = iamPolicyDocsMap
sys.iamUserPolicyMap = iamUserPolicyMap
sys.iamGroupPolicyMap = iamGroupPolicyMap
sys.iamGroupsMap = iamGroupsMap
sys.buildUserGroupMemberships()
return nil
}
func (iamOS *IAMObjectStore) savePolicyDoc(policyName string, p iampolicy.Policy) error {
return iamOS.saveIAMConfig(&p, getPolicyDocPath(policyName))
}
func (iamOS *IAMObjectStore) saveMappedPolicy(name string, isSTS, isGroup bool, mp MappedPolicy) error {
return iamOS.saveIAMConfig(mp, getMappedPolicyPath(name, isSTS, isGroup))
}
func (iamOS *IAMObjectStore) saveUserIdentity(name string, isSTS bool, u UserIdentity) error {
return iamOS.saveIAMConfig(u, getUserIdentityPath(name, isSTS))
}
func (iamOS *IAMObjectStore) saveGroupInfo(name string, gi GroupInfo) error {
return iamOS.saveIAMConfig(gi, getGroupInfoPath(name))
}
func (iamOS *IAMObjectStore) deletePolicyDoc(name string) error {
return iamOS.deleteIAMConfig(getPolicyDocPath(name))
}
func (iamOS *IAMObjectStore) deleteMappedPolicy(name string, isSTS, isGroup bool) error {
return iamOS.deleteIAMConfig(getMappedPolicyPath(name, isSTS, isGroup))
}
func (iamOS *IAMObjectStore) deleteUserIdentity(name string, isSTS bool) error {
return iamOS.deleteIAMConfig(getUserIdentityPath(name, isSTS))
}
func (iamOS *IAMObjectStore) deleteGroupInfo(name string) error {
return iamOS.deleteIAMConfig(getGroupInfoPath(name))
}
// helper type for listIAMConfigItems
type itemOrErr struct {
Item string
Err error
}
// Lists files or dirs in the minioMetaBucket at the given path
// prefix. If dirs is true, only directories are listed, otherwise
// only objects are listed. All returned items have the pathPrefix
// removed from their names.
func listIAMConfigItems(objectAPI ObjectLayer, pathPrefix string, dirs bool,
doneCh <-chan struct{}) <-chan itemOrErr {
ch := make(chan itemOrErr)
dirList := func(lo ListObjectsInfo) []string {
return lo.Prefixes
}
filesList := func(lo ListObjectsInfo) (r []string) {
for _, o := range lo.Objects {
r = append(r, o.Name)
}
return r
}
go func() {
marker := ""
for {
lo, err := objectAPI.ListObjects(context.Background(),
minioMetaBucket, pathPrefix, marker, SlashSeparator, 1000)
if err != nil {
select {
case ch <- itemOrErr{Err: err}:
case <-doneCh:
}
close(ch)
return
}
marker = lo.NextMarker
lister := dirList(lo)
if !dirs {
lister = filesList(lo)
}
for _, itemPrefix := range lister {
item := strings.TrimPrefix(itemPrefix, pathPrefix)
item = strings.TrimSuffix(item, SlashSeparator)
select {
case ch <- itemOrErr{Item: item}:
case <-doneCh:
close(ch)
return
}
}
if !lo.IsTruncated {
close(ch)
return
}
}
}()
return ch
}
func (iamOS *IAMObjectStore) watch(sys *IAMSys) {
watchDisk := func() {
ticker := time.NewTicker(globalRefreshIAMInterval)
defer ticker.Stop()
for {
select {
case <-GlobalServiceDoneCh:
return
case <-ticker.C:
iamOS.loadAll(sys, nil)
}
}
}
// Refresh IAMSys in background.
go watchDisk()
}
+714 -611
View File
File diff suppressed because it is too large Load Diff
+9
View File
@@ -50,6 +50,15 @@ func (sys *LifecycleSys) Set(bucketName string, lifecycle lifecycle.Lifecycle) {
sys.bucketLifecycleMap[bucketName] = lifecycle
}
// Get - gets lifecycle config associated to a given bucket name.
func (sys *LifecycleSys) Get(bucketName string) (lifecycle lifecycle.Lifecycle, ok bool) {
sys.Lock()
defer sys.Unlock()
l, ok := sys.bucketLifecycleMap[bucketName]
return l, ok
}
func saveLifecycleConfig(ctx context.Context, objAPI ObjectLayer, bucketName string, bucketLifecycle *lifecycle.Lifecycle) error {
data, err := xml.Marshal(bucketLifecycle)
if err != nil {
+8 -26
View File
@@ -17,10 +17,8 @@
package cmd
import (
"bytes"
"context"
"crypto/tls"
"encoding/gob"
"errors"
"io"
"sync"
@@ -131,32 +129,16 @@ func (client *lockRESTClient) Close() error {
// restCall makes a call to the lock REST server.
func (client *lockRESTClient) restCall(call string, args dsync.LockArgs) (reply bool, err error) {
values := url.Values{}
values.Set(lockRESTUID, args.UID)
values.Set(lockRESTSource, args.Source)
values.Set(lockRESTResource, args.Resource)
values.Set(lockRESTServerAddr, args.ServerAddr)
values.Set(lockRESTServerEndpoint, args.ServiceEndpoint)
reader := bytes.NewBuffer(make([]byte, 0, 2048))
err = gob.NewEncoder(reader).Encode(args)
if err != nil {
return false, err
}
respBody, err := client.call(call, nil, reader, -1)
if err != nil {
return false, err
}
var resp lockResponse
respBody, err := client.call(call, values, nil, -1)
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&resp)
if err != nil || !resp.Success {
reqInfo := &logger.ReqInfo{}
reqInfo.AppendTags("resource", args.Resource)
reqInfo.AppendTags("serveraddress", args.ServerAddr)
reqInfo.AppendTags("serviceendpoint", args.ServiceEndpoint)
reqInfo.AppendTags("source", args.Source)
reqInfo.AppendTags("uid", args.UID)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
return resp.Success, err
return err == nil, err
}
// RLock calls read lock REST API.
+13 -5
View File
@@ -21,7 +21,7 @@ import (
"time"
)
const lockRESTVersion = "v1"
const lockRESTVersion = "v2"
const lockRESTPath = minioReservedBucketPath + "/lock/" + lockRESTVersion
var lockServicePath = path.Join(minioReservedBucketPath, lockServiceSubPath)
@@ -33,6 +33,18 @@ const (
lockRESTMethodRUnlock = "runlock"
lockRESTMethodForceUnlock = "forceunlock"
lockRESTMethodExpired = "expired"
// Unique ID of lock/unlock request.
lockRESTUID = "uid"
// Source contains the line number, function and file name of the code
// on the client node that requested the lock.
lockRESTSource = "source"
// Resource contains a entity to be locked/unlocked.
lockRESTResource = "resource"
// ServerAddr contains the address of the server who requested lock/unlock of the above resource.
lockRESTServerAddr = "serverAddr"
// ServiceEndpoint contains the network path of above server to do lock/unlock.
lockRESTServerEndpoint = "serverEndpoint"
)
// nameLockRequesterInfoPair is a helper type for lock maintenance
@@ -41,10 +53,6 @@ type nameLockRequesterInfoPair struct {
lri lockRequesterInfo
}
type lockResponse struct {
Success bool
}
// Similar to removeEntry but only removes an entry only if the lock entry exists in map.
func (l *localLocker) removeEntryIfExists(nlrip nameLockRequesterInfoPair) {
// Check if entry is still in map (could have been removed altogether by 'concurrent' (R)Unlock of last entry)
+27 -117
View File
@@ -18,7 +18,6 @@ package cmd
import (
"context"
"encoding/gob"
"errors"
"math/rand"
"net/http"
@@ -60,6 +59,15 @@ func (l *lockRESTServer) IsValid(w http.ResponseWriter, r *http.Request) bool {
return true
}
func getLockArgs(r *http.Request) dsync.LockArgs {
return dsync.LockArgs{
UID: r.URL.Query().Get(lockRESTUID),
Resource: r.URL.Query().Get(lockRESTResource),
ServerAddr: r.URL.Query().Get(lockRESTServerAddr),
ServiceEndpoint: r.URL.Query().Get(lockRESTServerEndpoint),
}
}
// LockHandler - Acquires a lock.
func (l *lockRESTServer) LockHandler(w http.ResponseWriter, r *http.Request) {
if !l.IsValid(w, r) {
@@ -67,28 +75,10 @@ func (l *lockRESTServer) LockHandler(w http.ResponseWriter, r *http.Request) {
return
}
ctx := newContext(r, w, "Lock")
var lockArgs dsync.LockArgs
if r.ContentLength < 0 {
l.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&lockArgs)
if err != nil {
if _, err := l.ll.Lock(getLockArgs(r)); err != nil {
l.writeErrorResponse(w, err)
return
}
success, err := l.ll.Lock(lockArgs)
if err != nil {
l.writeErrorResponse(w, err)
return
}
resp := lockResponse{Success: success}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(resp))
w.(http.Flusher).Flush()
}
// UnlockHandler - releases the acquired lock.
@@ -98,28 +88,10 @@ func (l *lockRESTServer) UnlockHandler(w http.ResponseWriter, r *http.Request) {
return
}
ctx := newContext(r, w, "Unlock")
var lockArgs dsync.LockArgs
if r.ContentLength < 0 {
l.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&lockArgs)
if err != nil {
if _, err := l.ll.Unlock(getLockArgs(r)); err != nil {
l.writeErrorResponse(w, err)
return
}
success, err := l.ll.Unlock(lockArgs)
if err != nil {
l.writeErrorResponse(w, err)
return
}
resp := lockResponse{Success: success}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(resp))
w.(http.Flusher).Flush()
}
// LockHandler - Acquires an RLock.
@@ -129,27 +101,10 @@ func (l *lockRESTServer) RLockHandler(w http.ResponseWriter, r *http.Request) {
return
}
ctx := newContext(r, w, "RLock")
var lockArgs dsync.LockArgs
if r.ContentLength < 0 {
l.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&lockArgs)
if err != nil {
if _, err := l.ll.RLock(getLockArgs(r)); err != nil {
l.writeErrorResponse(w, err)
return
}
success, err := l.ll.RLock(lockArgs)
if err != nil {
l.writeErrorResponse(w, err)
return
}
resp := lockResponse{Success: success}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(resp))
w.(http.Flusher).Flush()
}
// RUnlockHandler - releases the acquired read lock.
@@ -159,27 +114,10 @@ func (l *lockRESTServer) RUnlockHandler(w http.ResponseWriter, r *http.Request)
return
}
ctx := newContext(r, w, "RUnlock")
var lockArgs dsync.LockArgs
if r.ContentLength < 0 {
l.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&lockArgs)
if err != nil {
if _, err := l.ll.RUnlock(getLockArgs(r)); err != nil {
l.writeErrorResponse(w, err)
return
}
success, err := l.ll.RUnlock(lockArgs)
if err != nil {
l.writeErrorResponse(w, err)
return
}
resp := lockResponse{Success: success}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(resp))
w.(http.Flusher).Flush()
}
// ForceUnlockHandler - force releases the acquired lock.
@@ -189,28 +127,10 @@ func (l *lockRESTServer) ForceUnlockHandler(w http.ResponseWriter, r *http.Reque
return
}
ctx := newContext(r, w, "ForceUnlock")
var lockArgs dsync.LockArgs
if r.ContentLength < 0 {
l.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&lockArgs)
if err != nil {
if _, err := l.ll.ForceUnlock(getLockArgs(r)); err != nil {
l.writeErrorResponse(w, err)
return
}
success, err := l.ll.ForceUnlock(lockArgs)
if err != nil {
l.writeErrorResponse(w, err)
return
}
resp := lockResponse{Success: success}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(resp))
w.(http.Flusher).Flush()
}
// ExpiredHandler - query expired lock status.
@@ -220,19 +140,8 @@ func (l *lockRESTServer) ExpiredHandler(w http.ResponseWriter, r *http.Request)
return
}
ctx := newContext(r, w, "Expired")
lockArgs := getLockArgs(r)
var lockArgs dsync.LockArgs
if r.ContentLength < 0 {
l.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&lockArgs)
if err != nil {
l.writeErrorResponse(w, err)
return
}
success := true
l.ll.mutex.Lock()
defer l.ll.mutex.Unlock()
@@ -246,11 +155,10 @@ func (l *lockRESTServer) ExpiredHandler(w http.ResponseWriter, r *http.Request)
}
}
}
// When we get here lock is no longer active due to either dsync.LockArgs.Resource
// being absent from map or uid not found for given dsync.LockArgs.Resource
resp := lockResponse{Success: success}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(resp))
w.(http.Flusher).Flush()
if !success {
l.writeErrorResponse(w, errors.New("lock already expired"))
return
}
}
// lockMaintenance loops over locks that have been active for some time and checks back
@@ -323,12 +231,14 @@ func startLockMaintenance(lkSrv *lockRESTServer) {
// registerLockRESTHandlers - register lock rest router.
func registerLockRESTHandlers(router *mux.Router) {
subrouter := router.PathPrefix(lockRESTPath).Subrouter()
subrouter.Methods(http.MethodPost).Path("/" + lockRESTMethodLock).HandlerFunc(httpTraceHdrs(globalLockServer.LockHandler))
subrouter.Methods(http.MethodPost).Path("/" + lockRESTMethodRLock).HandlerFunc(httpTraceHdrs(globalLockServer.RLockHandler))
subrouter.Methods(http.MethodPost).Path("/" + lockRESTMethodUnlock).HandlerFunc(httpTraceHdrs(globalLockServer.UnlockHandler))
subrouter.Methods(http.MethodPost).Path("/" + lockRESTMethodRUnlock).HandlerFunc(httpTraceHdrs(globalLockServer.RUnlockHandler))
subrouter.Methods(http.MethodPost).Path("/" + lockRESTMethodForceUnlock).HandlerFunc(httpTraceHdrs(globalLockServer.ForceUnlockHandler))
subrouter.Methods(http.MethodPost).Path("/" + lockRESTMethodExpired).HandlerFunc(httpTraceAll(globalLockServer.ExpiredHandler))
queries := restQueries(lockRESTUID, lockRESTSource, lockRESTResource, lockRESTServerAddr, lockRESTServerEndpoint)
subrouter.Methods(http.MethodPost).Path(SlashSeparator + lockRESTMethodLock).HandlerFunc(httpTraceHdrs(globalLockServer.LockHandler)).Queries(queries...)
subrouter.Methods(http.MethodPost).Path(SlashSeparator + lockRESTMethodRLock).HandlerFunc(httpTraceHdrs(globalLockServer.RLockHandler)).Queries(queries...)
subrouter.Methods(http.MethodPost).Path(SlashSeparator + lockRESTMethodUnlock).HandlerFunc(httpTraceHdrs(globalLockServer.UnlockHandler)).Queries(queries...)
subrouter.Methods(http.MethodPost).Path(SlashSeparator + lockRESTMethodRUnlock).HandlerFunc(httpTraceHdrs(globalLockServer.RUnlockHandler)).Queries(queries...)
subrouter.Methods(http.MethodPost).Path(SlashSeparator + lockRESTMethodForceUnlock).HandlerFunc(httpTraceHdrs(globalLockServer.ForceUnlockHandler)).Queries(queries...)
subrouter.Methods(http.MethodPost).Path(SlashSeparator + lockRESTMethodExpired).HandlerFunc(httpTraceAll(globalLockServer.ExpiredHandler)).Queries(queries...)
router.NotFoundHandler = http.HandlerFunc(httpTraceAll(notFoundHandler))
// Start lock maintenance from all lock servers.

Some files were not shown because too many files have changed in this diff Show More