Compare commits

..

159 Commits

Author SHA1 Message Date
Krishna Srinivas a2e904b966 Support any string as delimiter for listing (#7882) 2019-07-05 14:06:12 -07:00
Kaan Kabalak cc7dc61eb4 Allow folders inside buckets to be opened in a new tab (#7840)
Fixes #7836
2019-07-05 13:21:06 -07:00
Yao Zongyou c4f480a839 fix csv read bug (#7885) 2019-07-05 12:08:56 -07:00
Yao Zongyou 60831e3299 aggregation functions' argument may already has been cast to numeric (#7876) 2019-07-05 10:38:38 -07:00
Yao Zongyou 037319066f fix unicode support related bugs in s3select (#7877) 2019-07-05 09:43:10 -07:00
Praveen raj Mani bb871a7c31 Enable event persistence in webhook (#7614) 2019-07-05 15:21:41 +05:30
Harshavardhana 0ebbd3caef Avoid chown instead fallback to rootpath for user perms (#7874)
Fixes #7864
2019-07-03 18:57:34 -07:00
Ryan Tam bd56f80250 Fix ignored alias for aggregate result in S3 Select (#7849)
The SQL parser as it stands right now ignores alias for aggregate
result, e.g. `SELECT COUNT(*) AS thing FROM s3object` doesn't actually
return record like `{"thing": 42}`, it returns a record like `{"_1": 42}`.
Column alias for aggregate result is supported in AWS's S3 Select, so
this commit fixes that by respecting the `expr.As` in the expression.

Also improve test for S3 select

On top of testing a simple `SELECT` query, we want to test a few more
"advanced" queries (e.g. aggregation).

Convert existing tests into table driven tests[1], and add the new test
cases with "advanced" queries into them.

[1] - https://github.com/golang/go/wiki/TableDrivenTests
2019-07-03 16:34:54 -07:00
iliul a39e810965 docs: Fix dead link of HighwayHash (#7847)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-07-03 14:32:58 -07:00
mizuno-keyence 09103991ea [Bugfix] duplicating flag registration (#7853) 2019-07-03 14:31:19 -07:00
Harshavardhana c43f745449 Ensure that we use constants everywhere (#7845)
This allows for canonicalization of the strings
throughout our code and provides a common space
for all these constants to reside.

This list is rather non-exhaustive but captures
all the headers used in AWS S3 API operations
2019-07-02 22:34:32 -07:00
Anis Elleuch 9610a74c19 auto-heal: Use fast scan instead of the deep one (#7868) 2019-07-02 18:53:08 -07:00
Matthew Wegner 0bcd8abc5c doc: "admin user policy" command typo (#7865)
Under "change user policy", the `mc admin set-policy` command is wrong.  It should be `mc admin user set-policy`.
2019-07-02 11:48:26 -07:00
kannappanr 70b350c383 Remove DeploymentID from response headers (#7815)
Response headers need not contain deployment ID.
2019-07-01 12:22:01 -07:00
Krishna Srinivas 338e9a9be9 Put object client disconnect (#7824)
Fail putObject  and postpolicy in case client prematurely disconnects
Use request's context to cancel lock requests on client disconnects
2019-06-28 22:09:17 -07:00
Krishnan Parthasarathi edbd8709ec Simplify PR template to ease new contributors' workflow (#7844) 2019-06-27 15:47:46 -07:00
Minio Trusted 5db60a6c59 Update yaml files to latest version RELEASE.2019-06-27T21-13-50Z 2019-06-27 21:18:52 +00:00
Kaan Kabalak 36c19f1d65 Check bucket & prefix after obj's for new bucket are fetched (#7838)
Fixes #7837
2019-06-27 12:13:18 -07:00
iliul d3f9f8be88 golint: fix redundant code logic (#7842)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-06-27 15:18:33 +05:30
Krishna Srinivas 183ec094c4 Simplify HTTP trace related code (#7833) 2019-06-26 22:41:12 -07:00
Harshavardhana c1d2b3d5c3 Handle HEAD/GET requests for virtual DNS requests (#7839)
r.URL.Path is empty when HEAD bucket with virtual
DNS requests come in since bucket is now part of
r.Host, we should use our domain names and fetch
the right bucket/object names.

This fixes an really old issue in our federation
setups.
2019-06-26 18:21:54 -07:00
Praveen raj Mani be72609d1f Expose version info in prometheus (#7812)
Fixes #7795
2019-06-26 10:36:54 -07:00
Anis Elleuch 48f2c98052 admin: Add Background heal status info API (#7774)
This API returns the information related to the self healing routine.

For the moment, it returns:
- The total number of objects that are scanned
- The last time when an item was scanned
2019-06-25 16:42:24 -07:00
Kanagaraj M 286c663495 list objects in browser ordered by last modified (#7805)
- return all objects in web-handlers listObjects response
- added local pagination to object list ui
- also fixed infinite loader and removed unused fields
2019-06-25 16:31:50 -07:00
Yao Zongyou 941fed8e4a s3Select: call Close on error to release the read lock (#7830) 2019-06-25 13:30:48 -07:00
Yao Zongyou 55092bede1 add timestamp compare support (#7832) 2019-06-25 11:05:37 -07:00
Yao Zongyou 90a3b830f4 fix typo and the string representation of the time.Time value (#7831) 2019-06-25 09:54:14 -07:00
Yao Zongyou 23b9df0694 Fix s3select TRIM function's nil pointer dereference bug (#7817) 2019-06-24 16:59:33 -07:00
Kanagaraj M 48cb271a46 include ip address while doing checkPortAvailability (#7818)
While checking for port availability, ip address should be included.
When a machine has multiple ip addresses, multiple minio instances
or some other applications can be run on same port but different
ip address.

Fixes #7685
2019-06-24 15:02:39 -07:00
Klaus Post 61229b38f7 Update compression README.md (#7823)
- Snappy is not and RLE compressor, it is LZ77 based.
- Add `xz` as a common file type.
- Add most common media container types.
- Never heard of `application/x-spoon`. Google turns up a blank as well.
- Change link to minio blog post on compression & encryption.
2019-06-23 13:52:58 +05:30
Harshavardhana 90ca73af13 Allow trace even if server is not initialized (#7822) 2019-06-21 16:47:51 -07:00
Daniel Valdivia a04b6561a0 Fix a typo on the comment for ListenBucketNotification (#7821) 2019-06-21 11:58:52 -07:00
Andreas Auernhammer 219d841496 remove encryption key section for certtool docs (#7820)
This commit removes the encryption key section from
the certool.exe docs because:
 - MinIO does not support any TLS cipher that encrypts
   something with the private key. We only support PFS
   ciphers.
 - The doc comment is not really accurate anyway.
2019-06-21 10:39:02 -07:00
dependabot[bot] baef49b4a2 Bump stringstream from 0.0.5 to 0.0.6 in /browser (#7816)
Bumps [stringstream](https://github.com/mhart/StringStream) from 0.0.5 to 0.0.6.
- [Release notes](https://github.com/mhart/StringStream/releases)
- [Commits](https://github.com/mhart/StringStream/compare/v0.0.5...v0.0.6)

Signed-off-by: dependabot[bot] <support@github.com>
2019-06-21 07:49:51 -07:00
Ashish Kumar Sinha 3022f60561 Stop duplicate entry in Notification.xml (#7690)
fixes #7671
2019-06-21 13:28:02 +05:30
poornas 680fdf6f90 Update doc link (#7814) 2019-06-20 16:49:21 -07:00
Harshavardhana 1af6e8cb72 Add support for session policies in STS APIs (#7747)
This PR adds support for adding session policies
for further restrictions on STS credentials, useful
in situations when applications want to generate
creds for multiple interested parties with different
set of policy restrictions.

This session policy is not mandatory, but optional.

Fixes #7732
2019-06-20 15:28:33 -07:00
Andreas Auernhammer 98d3913a1e enable SSE-KMS pass-through on S3 gateway (#7788)
This commit relaxes the restriction that the MinIO gateway
does not accept SSE-KMS headers. Now, the S3 gateway allows
SSE-KMS headers for PUT and MULTIPART PUT requests and forwards them
to the S3 gateway backend (AWS). This is considered SSE pass-through
mode.

Fixes #7753
2019-06-19 17:37:08 -07:00
Harshavardhana 35c38e4bd8 Add docs regarding trace (#7808) 2019-06-19 14:28:03 -07:00
Minio Trusted e43d3a075c Update yaml files to latest version RELEASE.2019-06-19T18-24-42Z 2019-06-19 18:31:37 +00:00
Harshavardhana 43e0ef4248 Update docs with new settings (#7803) 2019-06-18 17:52:27 -07:00
Harshavardhana cd7d5b59e5 Add DeleteUser() to generate events in etcd (#7804)
Fixes a regression introduced in 6d89435356

Fixes #7797
2019-06-18 15:44:23 -07:00
poornas 299ef9b188 Trace: Replace function name with API prefix (#7794)
This change is required for `Admin Trace`
2019-06-18 13:55:13 -07:00
Harshavardhana b30c436715 [notify] Make sure to return when quorum is missing (#7799)
Fixes a regression introduced in 510ec153b9
2019-06-18 09:23:33 -07:00
Nitish Tiwari 7001fe407f Check local address in healthcheck script to ensure correct MinIO port (#7787)
Fixes #7780
2019-06-17 00:12:36 -07:00
Minio Trusted 59f7266081 Update yaml files to latest version RELEASE.2019-06-15T23-07-18Z 2019-06-15 23:17:28 +00:00
Harshavardhana 99bf4d0c42 [sec] Match ${aws:username} exactly instead of prefix match (#7791)
This PR fixes a security issue where an IAM user based
on his policy is granted more privileges than restricted
by the users IAM policy.

This is due to an issue of prefix based Matcher() function
which was incorrectly matching prefix based on resource
prefixes instead of exact match.
2019-06-15 11:27:17 -07:00
Cody Maloney 7b8beecc81 Move lock to not surround pieces which don't use any internal members. (#7779)
Previously the read/write lock applied both for gateway use cases as
well the object store use case. Nothing from sys is touched or looked
at in the gateway usecase though, so we don't need to lock. Don't lock
to make the gateway policy getting a little more efficient, particularly
as where this is called from (checkRequestAuthType) is quite common.
2019-06-15 10:11:10 -07:00
Praveen raj Mani 510ec153b9 Refreshing notification system should not erase the rules-map of other buckets (#7758)
Fixes #7707
2019-06-15 03:14:27 -07:00
dependabot[bot] e29a37e95c Bump macaddress from 0.2.8 to 0.2.9 in /browser (#7781)
Bumps [macaddress](https://github.com/scravy/node-macaddress) from 0.2.8 to 0.2.9.
- [Release notes](https://github.com/scravy/node-macaddress/releases)
- [Commits](https://github.com/scravy/node-macaddress/compare/0.2.8...0.2.9)

Signed-off-by: dependabot[bot] <support@github.com>
2019-06-15 03:07:56 -07:00
Harshavardhana 4a4048fe27 Migrate minio etcd config to backend config (#7751)
etcd when used in federated setups, currently
mandates that all clusters should have same
config.json, which is too restrictive and makes
federation a restrictive environment.

This change makes it apparent that each cluster
needs to be independently managed if necessary
from `mc admin info` command line.

Each cluster with in federation can have their
own root credentials and as well as separate
regions. This way buckets get further restrictions
and allows for root creds to be not common
across clusters/data centers.

Existing data in etcd gets migrated to backend
on each clusters, upon start. Once done
users can change their config entries
independently.
2019-06-15 03:07:54 -07:00
Minio Trusted da2887f914 Update yaml files to latest version RELEASE.2019-06-13T01-41-13Z 2019-06-13 01:46:00 +00:00
Harshavardhana c22439c82e Update minio-go v6.0.29 (#7778)
Bring improved retry logic
2019-06-12 18:09:21 -07:00
Harshavardhana 38224a4c1a Ignore errors reading fs.json (#7777) 2019-06-12 16:42:03 -07:00
Harshavardhana b4ab778cb2 Fix user IAM policy regression, reload policy appropriately (#7770)
Introduce in commit 7e4c9a9e1e

Fixes #7769
2019-06-12 14:49:45 -07:00
Harshavardhana 22f6756ce6 Allow su-exec to fail when users explicity use --user (#7776)
This allows MinIO containers to run properly without
expecting higher privileges in situations where following
restrictions on containers are used

 - docker run --user uid:gid
 - docker-compose up (with docker-compose.yml with user)
 ```yml
 ...
 user: "1001:1001"
 command: minio server /data
 ...
 ```
 - All openshift containers

Fixes #7773
2019-06-12 12:16:21 -07:00
Krishna Srinivas 0394a8f013 Send Content-Length in the response headers (#7771)
curl using http1.0 would hang sometimes when Content-Length is missing in response headers
fixes #7661
2019-06-11 21:04:52 -07:00
Harshavardhana 1039311184 Update UI assets (#7764) 2019-06-11 15:31:10 -07:00
Harshavardhana a075015293 doc: Merge large bucket with distributed docs (#7761) 2019-06-11 13:44:33 -07:00
Nitish Tiwari d90d4841b8 Add su-exec download for changing to non-root user (#7768)
Fixes #7767
2019-06-11 09:32:16 -07:00
Minio Trusted d3a2efbf91 Update yaml files to latest version RELEASE.2019-06-11T00-44-33Z 2019-06-11 00:50:31 +00:00
Harshavardhana 002a205c9c Fix OPA result response handling (#7763)
Also update the document with updated rego policy
and updated OPA agent REST API.

This PR is to fix a regression caused by PR #7637
2019-06-10 17:06:32 -07:00
Harshavardhana 91ceae23d0 Add support for customizable user (#7569) 2019-06-10 20:27:42 +05:30
kannappanr 1008c2c069 Do not display error logs if user does not have listbuckets privilege (#7370)
Fixes #7367
2019-06-09 13:15:57 -07:00
Anis Elleuch 7abadfccc2 Add self-healing feature (#7604)
- Background Heal routine receives heal requests from a channel, either to
heal format, buckets or objects
- Daily sweeper lists all objects in all buckets, these objects
don't necessarly have read quorum so they can be removed if
these objects are unhealable
- Heal daily ops receives objects from the daily sweeper
and send them to the heal routine.
2019-06-08 22:14:07 -07:00
poornas 97090aa16c Add admin API to send trace notifications to registered (#7128)
Remove current functionality to log trace to file
using MINIO_HTTP_TRACE env, and replace it with
mc admin trace command on mc client.
2019-06-08 15:54:41 -07:00
dependabot[bot] fb531235de Bump nwmatcher from 1.4.3 to 1.4.4 in /browser (#7760)
Bumps [nwmatcher](https://github.com/dperini/nwmatcher) from 1.4.3 to 1.4.4.
- [Release notes](https://github.com/dperini/nwmatcher/releases)
- [Commits](https://github.com/dperini/nwmatcher/commits)

Signed-off-by: dependabot[bot] <support@github.com>
2019-06-07 23:41:41 -07:00
Harshavardhana cb1566c6e6 S3 Gateway: Handle restricted access credentials (#7757) 2019-06-07 15:49:13 -07:00
Harshavardhana 6d89435356 Reload a specific user or policy on peers (#7705)
Fixes #7587
2019-06-06 17:46:22 -07:00
Andreas Auernhammer 975237cbf8 Create SECURITY.md (#7692) 2019-06-06 16:56:43 -07:00
dependabot[bot] 501af06001 Bump handlebars from 4.0.11 to 4.1.2 in /browser (#7755)
Bumps [handlebars](https://github.com/wycats/handlebars.js) from 4.0.11 to 4.1.2.
- [Release notes](https://github.com/wycats/handlebars.js/releases)
- [Changelog](https://github.com/wycats/handlebars.js/blob/master/release-notes.md)
- [Commits](https://github.com/wycats/handlebars.js/compare/v4.0.11...v4.1.2)

Signed-off-by: dependabot[bot] <support@github.com>
2019-06-06 11:16:09 -07:00
Harshavardhana a69f74533c Add region as part of error XML (#7752) 2019-06-05 16:28:21 -07:00
Joe Stevens a19cf063b5 Fixes for multiplatform dev and testing from forks (#7734)
Add support for correct dependency URLs on all platforms

only build mountinfo.go on linux

make testfile path relative to support fork work
2019-06-04 00:59:40 -07:00
Waldemar Quevedo 8b0c86298d Update import for NATS clients (#7743)
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
2019-06-04 11:16:07 +05:30
Minio Trusted df19546cc8 Update yaml files to latest version RELEASE.2019-06-04T01-15-58Z 2019-06-04 01:22:32 +00:00
Harshavardhana ff7799452b Fix build failure due to nats project migration 2019-06-03 17:49:49 -07:00
Harshavardhana 97be455f63 Fix build failure in web-handlers.go 2019-06-03 16:44:09 -07:00
Harshavardhana d16a409943 Ignore srvRecords from domain level entries (#7742)
Fixes #7741
2019-06-03 16:16:05 -07:00
Krishnan Parthasarathi 74efbb4153 Add deploymentID to web handler logs (#7712) 2019-06-03 15:40:04 -07:00
Harshavardhana 0cfd5a21ba [gateway] Remove policy reload, instead read policy from backend (#7727)
Inconsistencies can arise after applying bucket policies in
gateway mode, since all gateway instances do not share a
common shared state. This is by design to keep gateway as
shared nothing architecture.

This PR fixes such inconsistencies by reloading policy
if any from the backend.

Fixes #7723
2019-06-03 11:06:13 -07:00
dependabot[bot] 1ce2d29bbb Bump fstream from 1.0.11 to 1.0.12 in /browser (#7730)
Bumps [fstream](https://github.com/npm/fstream) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/npm/fstream/releases)
- [Commits](https://github.com/npm/fstream/compare/v1.0.11...v1.0.12)
2019-05-31 22:11:38 -07:00
Minio Trusted 48ffb7731a Update yaml files to latest version RELEASE.2019-06-01T03-46-14Z 2019-06-01 03:52:47 +00:00
Harshavardhana 1cfd4a48d9 Add specific headers in CORS, along with wildcard (#7726)
Fixes #7492
2019-05-31 09:23:55 -07:00
Harshavardhana 993a79d9c6 Disable http2 until we have upstream bugs fixed (#7711)
We should revert this PR in future once we
have upstream bugs fixed regarding http2 behavior
2019-05-30 19:49:33 -07:00
Harshavardhana 0c16b1c9a7 Fix docs.min.io linking issues (#7710) 2019-05-30 17:19:25 -07:00
Kanagaraj M 900cc27b51 validate keys before updating for IAM user (#7720)
New secretkey should be validated before updating
it on the config.

Fixes #7715
2019-05-30 05:14:35 -07:00
Kanagaraj M 8528017ad3 add min length validation to access key and secret key (#7721)
While changing the credentials through MinIO browser,
Update button will be disabled if keys are lesser than
minimum length.

Fixes #7713
2019-05-30 13:58:22 +05:30
Praveen raj Mani a73da7755e Remove senstive encryption entries from event data (#7719)
Fixes #7716
2019-05-29 22:29:37 -07:00
Harshavardhana 2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
Praveen raj Mani 763fce909b Enable event persistence in kafka (#7633) 2019-05-29 13:19:48 -07:00
Kanagaraj M da8214845a allow users to change password through browser (#7683)
Allow IAM users to change the password using
browser UI.
2019-05-29 13:18:46 -07:00
Krishna Srinivas 74e2fe0879 Return "SlowDown" to S3 clients for network related errors (#7610)
Consider errors returned by httpClient.Do() as network errors. This is because
the http clients returns different types of errors and it is hard to catch
all the error types.
2019-05-29 10:21:47 -07:00
Harshavardhana cb7f9ba286 Add disaggregated architecture document (#7691)
This document describes how to configure Spark,
Hive to use MinIO as replacement for HDFS.
2019-05-29 08:04:51 +05:30
Harshavardhana 7906f464fd Skip non-bucket dns entry (#7699)
Fixes #7697
2019-05-28 19:13:49 -07:00
Harshavardhana 7e4c9a9e1e Properly watch for users, policies, temp users (#7701)
Users were not reloaded properly when etcd was
configured in gateway, server modes.

This PR fixes this issue.
2019-05-28 11:18:53 +05:30
Scott Edlund f6fd407e47 Add limitation to b2.md (#7680)
The B2 gateway does not implement reporting md5sum as an etag response from PutObject.  Add to Known Limitations.
2019-05-27 21:04:43 -07:00
Nitish Tiwari 46ced81f41 Fix Gateway startup sequence to populate etcd (if set) with bucket info (#7686) 2019-05-24 08:41:52 +05:30
Minio Trusted 7a02faab72 Update yaml files to latest version RELEASE.2019-05-23T00-29-34Z 2019-05-23 00:35:13 +00:00
Dee Koder e252114f06 Revert "cache: Rewrite to cache only on download (#7575)" (#7684)
This reverts commit a13b58f630.
2019-05-22 14:54:15 -07:00
Harshavardhana 39b3e4f9b3 Avoid using io.ReadFull() for WriteAll and CreateFile (#7676)
With these changes we are now able to peak performances
for all Write() operations across disks HDD and NVMe.

Also adds readahead for disk reads, which also increases
performance for reads by 3x.
2019-05-22 13:47:15 -07:00
Anis Elleuch 158b8c2e86 sets: Correctly set IsTruncated in listing (#7675)
IsTruncated should not be set to true if there is no further
possible entries beyond maxKeys.

This commit will also move wide testing on object API from xl
to xl sets.
2019-05-22 13:36:16 -07:00
Praveen raj Mani c4c79f61ce Notification: Changes to persistent event store (#7658)
This patch includes the following changes in event store interface
- Removes memory store. We will not persist events in memory anymore, if `queueDir` is not set.
- Orders the events before replaying to the broker.
2019-05-22 13:34:48 -07:00
Harshavardhana 59e1d94770 Remove stale entry spurious logging (#7663)
The problem in current code was we were removing
an entry from a lock lockerMap without considering
the fact that different entry for same resource is
a possibility due the nature of locks that can be
acquired in parallel before we decide if the lock
is considered stale

A sequence of events is as follows

 - Lock("resource")
 - lockMaintenance(finds a long lived lock in this "resource")
 - Owner node rebooted which now retruns Expired() as true for
   this "resource"
 - Unlock("resource") which succeeded in quorum
 - Now by this time application retried and acquired a new
   Lock() on the same "resource"
 - Now that we have Expired() true from the previous call,
   we proceed to purge the entry from the local lockMap()
   local lockMap reports a different entry for the expired
   UID which results in a spurious log entry.

This PR removes this logging as this situation is an
expected scenario.
2019-05-22 12:21:36 -07:00
Andrei Mikhalenia 59e847aebe Signature v4: Allow signed headers from GET parameters 2019-05-21 21:00:02 -07:00
poornas a13b58f630 cache: Rewrite to cache only on download (#7575)
This will allow cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

Fixes: #7458, #7573, #6265, #6630
2019-05-22 08:30:27 +05:30
Kaan Kabalak 9e7a19d6b9 Fix object share link expiry always being 7 days (#7669)
The expiry value wasn't being sent to the server on modal value change

Fixes #7668
2019-05-19 17:45:28 -07:00
Harshavardhana 16c648b109 Remove "Connection" close instead reduce MaxConns per host (#7654)
This is necessary to avoid connection build up between servers
unexpectedly for example in a situation where 16 servers are
talking to each other and one server now allows a maximum of

15*4096 = 61440 idle connections

Will be kept in pool. Such a large pool is perhaps inefficient for
many reasons and also affects overall system resources.

This PR also reduces idleConnection timeout from 120 secs to 60 secs.
2019-05-17 12:52:25 +05:30
Krishnan Parthasarathi c871456269 File must be sync'd before closing (#7657)
- group sync and close action into a single defer statement to avoid
  evaluation order related bugs in future.
2019-05-16 18:30:51 -07:00
Harshavardhana 55aa20595f Remove an empty entry being added into XML marshal (#7656) 2019-05-16 13:09:21 -07:00
poornas 707ed2b302 gcs: use MD5Sum as ETag if present in object attrs (#7643)
Fixes: 7642
2019-05-16 12:00:12 -07:00
ebozduman 78be3f8947 Removes the incorrect coverage badge from the docs (#7651) 2019-05-16 12:11:49 +05:30
ebozduman 67d508b214 Adjusts help content dynamically according to OS (#7646) 2019-05-15 14:02:44 +05:30
Harshavardhana 0022c9d210 Add connection close proactively for Walk() http/rpc (#7645) 2019-05-14 16:10:51 -07:00
Anis Elleuch 9b4a81ee60 xl: Avoid possible race during bulk Multi Delete (#7644)
errs was passed to many goroutines but they are all allowed
to update errs if any error happens during deletion, which
can cause a data race.

This commit will avoid issuing bulk delete operations in parallel
to avoid the warning race.
2019-05-14 14:43:22 -07:00
Harshavardhana b3f22eac56 Offload listing to posix layer (#7611)
This PR adds one API WalkCh which
sorts and sends list over the network

Each disk walks independently in a sorted manner.
2019-05-14 13:49:10 -07:00
Krishna Srinivas a343d14f19 Simplify putObject by not breaking the stream into parts (#7199)
We broke into parts previously as we had checksum for the entire file
to tax less on memory and to have better TTFB. We dont need to now,
after the streaming-bitrot change.
2019-05-14 12:33:18 -07:00
Anis Elleuch 9c90a28546 Implement bulk delete (#7607)
Bulk delete at storage level in Multiple Delete Objects API

In order to accelerate bulk delete in Multiple Delete objects API,
a new bulk delete is introduced in storage layer, which will accept
a list of objects to delete rather than only one. Consequently,
a new API is also need to be added to Object API.
2019-05-13 12:25:49 -07:00
Praveen raj Mani d9a7f80f68 Remove duplicate checkPutObjectArgs in PutObject and (#7396)
Fixes #7384
2019-05-13 10:12:06 -07:00
Krishna Srinivas 502456db91 Accept simpler response from OPA (#7637) 2019-05-12 18:40:49 +05:30
Harshavardhana 3eb7a8bde8 Sync before Close() to avoid random I/O (#7638) 2019-05-11 15:03:10 -07:00
Harshavardhana ac3b59645e Add API retry functionality in mc admin (#7602) 2019-05-09 17:41:54 -07:00
Harshavardhana 72929ec05b Turn off md5sum optionally if content-md5 is not set (#7609)
This PR also brings --compat option to run MinIO in strict
S3 compatibility mode, MinIO by default will now try to run
high performance mode.
2019-05-08 18:35:40 -07:00
Harshavardhana 757c639044 Separate issue templates for bug report and feature requests (#7616) 2019-05-07 13:49:54 -07:00
Aditya Manthramurthy 589df3d5e7 Deadcode removal (#7627) 2019-05-07 13:49:15 -07:00
Harshavardhana a71e08efb4 On macOS avoid using GNU specific options (#7621)
Fix Makefile from `cp -uf` to `cp -f`. We are only
interested in `-f` anyways.
2019-05-07 10:39:26 -07:00
kannappanr c422f7f412 Fix: Handle regression caused by gorilla mux v1.7.0 (#7625)
PR #7595 fixed part of the regression, but did not handle the
scenario, where in docker, the internal port is different from
the port on the host.

This PR modifies the regular expression such that all the
scenarios are handled.

Fixes #7619
2019-05-07 10:36:00 -07:00
Anis Elleuch 08b9244c48 Fix listing empty directory in recursive mode (#7613)
After recent listing refactor, recursive list doesn't return empty
directories, this commit will fix the behavior and add unit tests
so it won't happen again.
2019-05-06 07:52:42 -07:00
kannappanr efbc665ad3 Delete checked in binary file (#7615)
healthcheck binary was checked in by mistake.
2019-05-03 13:16:35 -07:00
Praveen raj Mani 998f01fadc Support IPv6 hosts for postgres connections (#7168)
Fixes #7145
2019-05-03 17:31:33 +05:30
kannappanr 4b858b562a Compression: Handle auto encryption when size is unknown (#7600)
When size is unknown and auto encryption is enabled,
and compression is set to true, putobject API is failing.

Moving adding the SSE-S3 header as part of the request to before
checking if compression can be done, otherwise the size is set to -1
and that seems to cause problems.
2019-05-02 08:28:18 -07:00
poornas 033f3a4d51 gcs: check error on object writer close (#7606)
Fixes #7605. Object metadata should be written to storage
only when the object was written successfully
2019-05-02 08:27:10 -07:00
poornas cf2a436bc8 Show SlowDown error message if backend is busy (#7521)
or if there are too many open file descriptors.
2019-05-02 07:09:57 -07:00
Harshavardhana 64998fc4ab Remove delayIsLeaf requirement simplify ListObjects further (#7593) 2019-05-02 10:36:57 +05:30
Krishna Srinivas 43be00ea1b Remove logs from bitrot-streaming.go as erasure layer is already logging (#7603) 2019-05-01 21:46:00 -07:00
Eco 3517f64d20 Clarify server maximums via Federation (#7608)
To eliminate confusion about 32 server limit, clarify that the limit is only per cluster, but that unlimited clusters can be Federated
2019-05-01 16:20:25 -07:00
Harshavardhana c5f26d5cdd Fix hdfsReader fd leak upon GetObject() (#7596)
Also migrate to minio/hdfs/v3@v3.0.0
2019-05-01 14:43:21 -07:00
Praveen raj Mani c113d4e49c Posix CreateFile should work for compressed lengths (#7584) 2019-04-30 16:27:31 -07:00
kannappanr a436f2baa5 Change order of trace source in error log (#7599)
Change the order of trace source that gets
printed on the console.
2019-04-29 14:56:30 -07:00
parnigot ba76cd3268 Minor TLS documentation improvements (#7592)
Added the explicit name for private.key and public.crt
2019-04-29 12:56:34 -07:00
kannappanr 781012517d Fix: Handle regression caused by gorilla mux v1.7.0 (#7595)
gorilla/mux#383 broke the compatibility with the existing code.
This PR handles that scenario.
2019-04-29 22:03:27 +05:30
Harshavardhana 091b9b661f Complain if we detect sub-optimal ordering in distributed setup (#7576)
Fixes #6156
2019-04-29 10:10:50 +05:30
Harshavardhana af6c6a2b35 Remove timeout conn on net.Dialer (#7590)
This PR also removes conn_bug_21133 workaround
which is not valid anymore, all we need is deadline
connection with server in place

Fixes #7503
2019-04-27 15:14:16 -07:00
Krishna Srinivas b93ef73f9b Fix divide by 0 error when directio.AlignSize is 0 (#7591) 2019-04-26 16:08:15 -07:00
Harshavardhana 83ca1a8d64 Use etcd watch to reload IAM users (#7551)
Currently we used to reload users every five minutes,
regardless of etcd is configured or not. But with etcd
configured we can do this more asynchronously to trigger
a refresh by using the watch API

Fixes #7515
2019-04-26 18:48:50 +05:30
Anis Elleuch 27ef1262bf xl: Use random UUID during complete multipart upload (#7527)
One user has seen this following error log:

API: CompleteMultipartUpload(bucket=vertica, object=perf-dss-v03/cc2/02596813aecd4e476d810148586c2a3300d00000013557ef_0.gt)
Time: 15:44:07 UTC 04/11/2019
RequestID: 159475EFF4DEDFFB
RemoteHost: 172.26.87.184
UserAgent: vertica-v9.1.1-5
Error: open /data/.minio.sys/tmp/100bb3ec-6c0d-4a37-8b36-65241050eb02/xl.json: file exists
       1: cmd/xl-v1-metadata.go:448:cmd.writeXLMetadata()
       2: cmd/xl-v1-metadata.go:501:cmd.writeUniqueXLMetadata.func1()

This can happen when CompleteMultipartUpload fails with write quorum,
the S3 client will retry (since write quorum is 500 http response),
however the second call of CompleteMultipartUpload will fail because
this latter doesn't truly use a random uuid under .minio.sys/tmp/
directory but pick the upload id.

This commit fixes the behavior to choose a random uuid for generating
xl.json
2019-04-25 07:33:26 -07:00
Harshavardhana ae002aa724 Deprecate updating admin credentials using API calls (#7570)
Root credentials are not allowed to change in all of our
distributed setup deployments, this PR simply removes
that behavior.
2019-04-24 12:54:44 -07:00
Krishna Srinivas a3ec71bc28 Use O_DIRECT while writing to disk (#7479)
- Use O_DIRECT while writing to disk
- Remove MINIO_DRIVE_SYNC option
2019-04-23 21:25:06 -07:00
Minio Trusted ab711fe1a2 Update yaml files to latest version RELEASE.2019-04-23T23-50-36Z 2019-04-23 23:55:31 +00:00
Harshavardhana 35d19a4ae2 Fix STS AssumeRole route conflict with MultipartUpload (#7574)
Since AssumeRole API was introduced we have a wrong route
match which results in certain clients failing to upload objects
using multipart because, multipart POST conflicts with STS POST
AssumeRole API.

Write a proper matcher function which verifies the route more
appropriately such that both can co-exist.
2019-04-23 15:55:41 -07:00
Harshavardhana f767a2538a Optimize listing with leaf check offloaded to posix (#7541)
Other listing optimizations include

- remove double sorting while filtering object entries
- improve error message when upload-id is not in quorum
- use jsoniter for full unmarshal json, instead of gjson
- remove unused code
2019-04-23 14:54:28 -07:00
kannappanr 0c75395abe Fix: Allow deleting multiple objects anonymously if policy supports it (#7439)
Fixes #5683
2019-04-22 20:24:43 +05:30
Roman Kalashnikov 188cf1d5ce Add more friendly error message for policy object (#7412) 2019-04-22 01:23:54 -07:00
Praveen raj Mani d96584ef58 Allow server to start if one of local nodes in docker/kubernetes setup is resolved (#7452)
Allow server to start if one of the local nodes in docker/kubernetes setup is successfully resolved

- The rule is that we need atleast one local node to work. We dont need to resolve the
  rest at that point.

- In a non-orchestrational setup, we fail if we do not have atleast one local node up
  and running.

- In an orchestrational setup (docker-swarm and kubernetes), We retry with a sleep of 5
  seconds until any one local node shows up.

Fixes #6995
2019-04-19 10:26:44 -07:00
Minio Trusted d42496cc74 Update yaml files to latest version RELEASE.2019-04-18T21-44-59Z 2019-04-18 21:52:22 +00:00
poornas 2c096c569f do not try to delete non-existent object in cache (#7560)
handle cache cleanup correctly when backend object was deleted.

Fixes: #7558
2019-04-18 13:53:22 -07:00
Chris Hoffman 816459d10f Azure gateway complete multipart ETag (#7500)
Compute md5 for azure multipart upload that matches s3 behavior
Reuse complete multipart md5 function in azure gateway
2019-04-17 23:50:25 -07:00
Kanagaraj M 3efcd6fa46 clear errors of previous failed login attempts (#7516)
When the first login attempt is failed(due to incorrect secretkey)
and the next attempt is successful. Error message shown for
the previous attempts should go away.

Fixes #7514
2019-04-18 11:49:51 +05:30
kannappanr d2f42d830f Lock: Use REST API instead of RPC (#7469)
In distributed mode, use REST API to acquire and manage locks instead
of RPC.

RPC has been completely removed from MinIO source.

Since we are moving from RPC to REST, we cannot use rolling upgrades as the
nodes that have not yet been upgraded cannot talk to the ones that have
been upgraded.

We expect all minio processes on all nodes to be stopped and then the
upgrade process to be completed.

Also force http1.1 for inter-node communication
2019-04-17 23:16:27 -07:00
Minio Trusted 7686340621 Update yaml files to latest version RELEASE.2019-04-18T01-15-57Z 2019-04-18 01:22:00 +00:00
339 changed files with 24979 additions and 7915 deletions
+46
View File
@@ -0,0 +1,46 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: community, triage
assignees: ''
---
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->
## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
1.
2.
3.
4.
## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio version`):
* Environment name and version (e.g. nginx 1.9.1):
* Server type and version:
* Operating System and version (`uname -a`):
* Link to your project:
+20
View File
@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: community, triage
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
+7 -21
View File
@@ -1,33 +1,19 @@
<!--- Provide a general summary of your changes in the Title above -->
## Description
<!--- Describe your changes in detail -->
## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->
## Regression
<!-- Is this PR fixing a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
## How to test this PR?
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have added unit tests to cover my changes.
- [ ] I have added/updated functional tests in [mint](https://github.com/minio/mint). (If yes, add `mint` PR # here: )
- [ ] All new and existing tests passed.
- [ ] Fixes a regression (If yes, please add `commit-id` or `PR #` here)
- [ ] Documentation needed
- [ ] Unit tests needed
- [ ] Functional tests needed (If yes, add [mint](https://github.com/minio/mint) PR # here: )
+1
View File
@@ -24,3 +24,4 @@ stage/
.sia_temp/
config.json
healthcheck
check-user
+1 -1
View File
@@ -1,4 +1,4 @@
# MinIO Contribution Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/) [![codecov](https://codecov.io/gh/minio/minio/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/minio)
# MinIO Contribution Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
``MinIO`` community welcomes your contribution. To make the process as seamless as possible, we recommend you read this contribution guide.
+4 -2
View File
@@ -10,7 +10,8 @@ RUN \
apk add --no-cache git && \
git clone https://github.com/minio/minio && cd minio && \
go install -v -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
cd dockerscripts; go build -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go
cd dockerscripts; go build -tags kqueue -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go && \
go build -tags kqueue -ldflags "-s -w" -o /usr/bin/check-user check-user.go
FROM alpine:3.9
@@ -22,10 +23,11 @@ EXPOSE 9000
COPY --from=0 /go/bin/minio /usr/bin/minio
COPY --from=0 /usr/bin/healthcheck /usr/bin/healthcheck
COPY --from=0 /usr/bin/check-user /usr/bin/check-user
COPY dockerscripts/docker-entrypoint.sh /usr/bin/
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' && \
apk add --no-cache ca-certificates 'curl>7.61.0' 'su-exec>=0.2' && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
+4 -3
View File
@@ -2,7 +2,7 @@ FROM alpine:3.9
LABEL maintainer="MinIO Inc <dev@min.io>"
COPY dockerscripts/docker-entrypoint.sh dockerscripts/healthcheck /usr/bin/
COPY dockerscripts/docker-entrypoint.sh dockerscripts/healthcheck dockerscripts/check-user /usr/bin/
COPY minio /usr/bin/
ENV MINIO_UPDATE off
@@ -10,11 +10,12 @@ ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' && \
apk add --no-cache ca-certificates 'curl>7.61.0' 'su-exec>=0.2' && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck
chmod +x /usr/bin/healthcheck && \
chmod +x /usr/bin/check-user
EXPOSE 9000
+6 -3
View File
@@ -7,13 +7,15 @@ ENV GO111MODULE on
RUN \
apk add --no-cache git && \
git clone https://github.com/minio/minio && cd minio/dockerscripts && \
go build -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go
go build -tags kqueue -ldflags "-s -w" -o /usr/bin/healthcheck healthcheck.go && \
go build -tags kqueue -ldflags "-s -w" -o /usr/bin/check-user check-user.go
FROM alpine:3.9
LABEL maintainer="MinIO Inc <dev@min.io>"
COPY --from=0 /usr/bin/healthcheck /usr/bin/healthcheck
COPY --from=0 /usr/bin/check-user /usr/bin/check-user
COPY dockerscripts/docker-entrypoint.sh /usr/bin/
ENV MINIO_UPDATE off
@@ -21,12 +23,13 @@ ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
RUN \
apk add --no-cache ca-certificates 'curl>7.61.0' && \
apk add --no-cache ca-certificates 'curl>7.61.0' 'su-exec>=0.2' && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
curl https://dl.min.io/server/minio/release/linux-amd64/minio > /usr/bin/minio && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck
chmod +x /usr/bin/healthcheck && \
chmod +x /usr/bin/check-user
EXPOSE 9000
+10 -3
View File
@@ -2,6 +2,12 @@ PWD := $(shell pwd)
GOPATH := $(shell go env GOPATH)
LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
GOOS := $(shell go env GOOS)
GOOSALT ?= 'linux'
ifeq ($(GOOS),'darwin')
GOOSALT = 'mac'
endif
TAG ?= $(USER)
BUILD_LDFLAGS := '$(LDFLAGS)'
@@ -14,8 +20,8 @@ checks:
getdeps:
@mkdir -p ${GOPATH}/bin
@which golint 1>/dev/null || (echo "Installing golint" && go get -u golang.org/x/lint/golint)
@which staticcheck 1>/dev/null || (echo "Installing staticcheck" && wget --quiet -O ${GOPATH}/bin/staticcheck https://github.com/dominikh/go-tools/releases/download/2019.1/staticcheck_linux_amd64 && chmod +x ${GOPATH}/bin/staticcheck)
@which misspell 1>/dev/null || (echo "Installing misspell" && wget --quiet https://github.com/client9/misspell/releases/download/v0.3.4/misspell_0.3.4_linux_64bit.tar.gz && tar xf misspell_0.3.4_linux_64bit.tar.gz && mv misspell ${GOPATH}/bin/misspell && chmod +x ${GOPATH}/bin/misspell && rm -f misspell_0.3.4_linux_64bit.tar.gz)
@which staticcheck 1>/dev/null || (echo "Installing staticcheck" && wget --quiet -O ${GOPATH}/bin/staticcheck https://github.com/dominikh/go-tools/releases/download/2019.1/staticcheck_${GOOS}_amd64 && chmod +x ${GOPATH}/bin/staticcheck)
@which misspell 1>/dev/null || (echo "Installing misspell" && wget --quiet https://github.com/client9/misspell/releases/download/v0.3.4/misspell_0.3.4_${GOOSALT}_64bit.tar.gz && tar xf misspell_0.3.4_${GOOSALT}_64bit.tar.gz && mv misspell ${GOPATH}/bin/misspell && chmod +x ${GOPATH}/bin/misspell && rm -f misspell_0.3.4_${GOOSALT}_64bit.tar.gz)
crosscompile:
@(env bash $(PWD)/buildscripts/cross-compile.sh)
@@ -67,6 +73,7 @@ build: checks
@echo "Building minio binary to './minio'"
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/minio 1>/dev/null
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/dockerscripts/healthcheck $(PWD)/dockerscripts/healthcheck.go 1>/dev/null
@GO111MODULE=on GOFLAGS="" CGO_ENABLED=0 go build -tags kqueue --ldflags $(BUILD_LDFLAGS) -o $(PWD)/dockerscripts/check-user $(PWD)/dockerscripts/check-user.go 1>/dev/null
docker: build
@docker build -t $(TAG) . -f Dockerfile.dev
@@ -74,7 +81,7 @@ docker: build
# Builds minio and installs it to $GOPATH/bin.
install: build
@echo "Installing minio binary to '$(GOPATH)/bin/minio'"
@mkdir -p $(GOPATH)/bin && cp -uf $(PWD)/minio $(GOPATH)/bin/minio
@mkdir -p $(GOPATH)/bin && cp -f $(PWD)/minio $(GOPATH)/bin/minio
@echo "Installation successful. To learn more, try \"minio --help\"."
clean:
+1 -1
View File
@@ -1,5 +1,5 @@
# MinIO Quickstart Guide
[![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/) [![codecov](https://codecov.io/gh/minio/minio/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/minio)
[![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
+1 -1
View File
@@ -1,4 +1,4 @@
# MinIO Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/) [![codecov](https://codecov.io/gh/minio/minio/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/minio)
# MinIO Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
MinIO 是一个基于Apache License v2.0开源协议的对象存储服务。它兼容亚马逊S3云存储服务接口,非常适合于存储大容量非结构化的数据,例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等,而一个对象文件可以是任意大小,从几kb到最大5T不等。
+41
View File
@@ -0,0 +1,41 @@
# Security Policy
## Supported Versions
We always provide security updates for the [latest release](https://github.com/minio/minio/releases/latest).
Whenever there is a security update you just need to upgrade to the latest version.
## Reporting a Vulnerability
All security bugs in [minio/minio](https://github,com/minio/minio) (or other minio/* repositories)
should be reported by email to security@min.io. Your email will be acknowledged within 48 hours,
and you'll receive a more detailed response to your email within 72 hours indicating the next steps
in handling your report.
Please, provide a detailed explanation of the issue. In particular, outline the type of the security
issue (DoS, authentication bypass, information disclouse, ...) and the assumptions you're making (e.g. do
you need access credentials for a successful exploit).
If you have not received a reply to your email within 48 hours or you have not heard from the security team
for the past five days please contact the security team directly:
- Primary security coordinator: aead@min.io
- Secondary coordinator: harsha@min.io
- If you receive no response: dev@min.io
### Disclosure Process
MinIO uses the following disclosure process:
1. Once the security report is received one member of the security team tries to verify and reproduce
the issue and determines the impact it has.
2. A member of the security team will respond and either confrim or reject the security report.
If the report is rejected the repsonse explains why.
3. Code is audited to find any potential similar problems.
4. Fixes are prepared for the latest release.
5. On the date that the fixes are applied a security advisory will be published on https://blog.min.io.
Please inform us in your report email whether MinIO should mention your contribution w.r.t. fixing
the security issue. By default MinIO will **not** publish this information to protect your privacy.
This process can take some time, especially when coordination is required with maintainers of other projects.
Every effort will be made to handle the bug in as timely a manner as possible, however it's important that we
follow the process described above to ensure that disclosures are handled consistently.
+9 -8
View File
@@ -4,10 +4,11 @@
## Installation
### Install yarn
### Install node
```sh
curl -o- -L https://yarnpkg.com/install.sh | bash
yarn
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
exec -l $SHELL
nvm install stable
```
### Install `go-bindata` and `go-bindata-assetfs`
@@ -15,8 +16,8 @@ yarn
If you do not have a working Golang environment, please follow [Install Golang](https://docs.min.io/docs/how-to-install-golang)
```sh
go get github.com/jteeuwen/go-bindata/...
go get github.com/elazarl/go-bindata-assetfs/...
go get github.com/go-bindata/go-bindata/go-bindata
go get github.com/elazarl/go-bindata-assetfs/go-bindata-assetfs
```
## Generating Assets
@@ -24,7 +25,7 @@ go get github.com/elazarl/go-bindata-assetfs/...
### Generate ui-assets.go
```sh
yarn release
npm run release
```
This generates ui-assets.go in the current directory. Now do `make` in the parent directory to build the minio binary with the newly generated ``ui-assets.go``
@@ -32,7 +33,7 @@ This generates ui-assets.go in the current directory. Now do `make` in the paren
### Run MinIO Browser with live reload
```sh
yarn dev
npm run dev
```
Open [http://localhost:8080/minio/](http://localhost:8080/minio/) in your browser to play with the application
@@ -66,7 +67,7 @@ index 3ccdaba..9496c56 100644
```
```sh
yarn dev
npm run dev
```
Open [http://localhost:8888/minio/](http://localhost:8888/minio/) in your browser to play with the application
+192 -109
View File
@@ -18,157 +18,237 @@ import React from "react"
import { connect } from "react-redux"
import web from "../web"
import * as alertActions from "../alert/actions"
import { getRandomAccessKey, getRandomSecretKey } from "../utils"
import jwtDecode from "jwt-decode"
import classNames from "classnames"
import {
Tooltip,
Modal,
ModalBody,
ModalHeader,
OverlayTrigger
} from "react-bootstrap"
import { Modal, ModalBody, ModalHeader } from "react-bootstrap"
import InputGroup from "./InputGroup"
import { ACCESS_KEY_MIN_LENGTH, SECRET_KEY_MIN_LENGTH } from "../constants"
export class ChangePasswordModal extends React.Component {
constructor(props) {
super(props)
this.state = {
accessKey: "",
secretKey: "",
keysReadOnly: false
currentAccessKey: "",
currentSecretKey: "",
currentSecretKeyVisible: false,
newAccessKey: "",
newSecretKey: "",
newSecretKeyVisible: false
}
}
// When its shown, it loads the access key and secret key.
// When its shown, it loads the access key from JWT token
componentWillMount() {
const { serverInfo } = this.props
// Check environment variables first.
if (serverInfo.info.isEnvCreds || serverInfo.info.isWorm) {
this.setState({
accessKey: "xxxxxxxxx",
secretKey: "xxxxxxxxx",
keysReadOnly: true
})
} else {
web.GetAuth().then(data => {
this.setState({
accessKey: data.accessKey,
secretKey: data.secretKey
})
})
}
}
// Handle field changes from inside the modal.
accessKeyChange(e) {
const token = jwtDecode(web.GetToken())
this.setState({
accessKey: e.target.value
})
}
secretKeyChange(e) {
this.setState({
secretKey: e.target.value
})
}
secretKeyVisible(secretKeyVisible) {
this.setState({
secretKeyVisible
currentAccessKey: token.sub,
newAccessKey: token.sub
})
}
// Save the auth params and set them.
setAuth(e) {
const { showAlert } = this.props
const accessKey = this.state.accessKey
const secretKey = this.state.secretKey
web
.SetAuth({
accessKey,
secretKey
})
.then(data => {
showAlert({
type: "success",
message: "Changed credentials"
if (this.canUpdateCredentials()) {
const currentAccessKey = this.state.currentAccessKey
const currentSecretKey = this.state.currentSecretKey
const newAccessKey = this.state.newAccessKey
const newSecretKey = this.state.newSecretKey
web
.SetAuth({
currentAccessKey,
currentSecretKey,
newAccessKey,
newSecretKey
})
})
.catch(err => {
showAlert({
type: "danger",
message: err.message
.then(data => {
showAlert({
type: "success",
message: "Credentials updated successfully."
})
})
})
.catch(err => {
showAlert({
type: "danger",
message: err.message
})
})
}
}
generateAuth(e) {
web.GenerateAuth().then(data => {
const { serverInfo } = this.props
// Generate random access key only for root user
if (!serverInfo.userInfo.isIAMUser) {
this.setState({
accessKey: data.accessKey,
secretKey: data.secretKey,
secretKeyVisible: true
newAccessKey: getRandomAccessKey()
})
}
this.setState({
newSecretKey: getRandomSecretKey(),
newSecretKeyVisible: true
})
}
canChangePassword() {
const { serverInfo } = this.props
// Password change is not allowed in WORM mode
if (serverInfo.info.isWorm) {
return false
}
// When credentials are set on ENV, password change not allowed for owner
if (serverInfo.info.isEnvCreds && !serverInfo.userInfo.isIAMUser) {
return false
}
return true
}
canUpdateCredentials() {
return (
this.state.currentAccessKey.length > 0 &&
this.state.currentSecretKey.length > 0 &&
this.state.newAccessKey.length >= ACCESS_KEY_MIN_LENGTH &&
this.state.newSecretKey.length >= SECRET_KEY_MIN_LENGTH
)
}
render() {
const { hideChangePassword } = this.props
const { hideChangePassword, serverInfo } = this.props
const allowChangePassword = this.canChangePassword()
if (!allowChangePassword) {
return (
<Modal bsSize="sm" animation={false} show={true}>
<ModalHeader>Change Password</ModalHeader>
<ModalBody>
Credentials of this user cannot be updated through MinIO Browser.
</ModalBody>
<div className="modal-footer">
<button
id="cancel-change-password"
className="btn btn-link"
onClick={hideChangePassword}
>
Close
</button>
</div>
</Modal>
)
}
return (
<Modal bsSize="sm" animation={false} show={true}>
<ModalHeader>Change Password</ModalHeader>
<ModalBody className="m-t-20">
<InputGroup
value={this.state.accessKey}
onChange={this.accessKeyChange.bind(this)}
id="accessKey"
label="Access Key"
name="accesskey"
type="text"
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
readonly={this.state.keysReadOnly}
/>
<i
onClick={this.secretKeyVisible.bind(
this,
!this.state.secretKeyVisible
<div className="has-toggle-password">
<InputGroup
value={this.state.currentAccessKey}
id="currentAccessKey"
label="Current Access Key"
name="currentAccesskey"
type="text"
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
readonly={true}
/>
<i
onClick={() => {
this.setState({
currentSecretKeyVisible: !this.state.currentSecretKeyVisible
})
}}
className={
"toggle-password fa fa-eye " +
(this.state.currentSecretKeyVisible ? "toggled" : "")
}
/>
<InputGroup
value={this.state.currentSecretKey}
onChange={e => {
this.setState({ currentSecretKey: e.target.value })
}}
id="currentSecretKey"
label="Current Secret Key"
name="currentSecretKey"
type={this.state.currentSecretKeyVisible ? "text" : "password"}
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
/>
</div>
<div className="has-toggle-password m-t-30">
{!serverInfo.userInfo.isIAMUser && (
<InputGroup
value={this.state.newAccessKey}
id="newAccessKey"
label={"New Access Key"}
name="newAccesskey"
type="text"
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
onChange={e => {
this.setState({ newAccessKey: e.target.value })
}}
readonly={serverInfo.userInfo.isIAMUser}
/>
)}
className={
"toggle-password fa fa-eye " +
(this.state.secretKeyVisible ? "toggled" : "")
}
/>
<InputGroup
value={this.state.secretKey}
onChange={this.secretKeyChange.bind(this)}
id="secretKey"
label="Secret Key"
name="accesskey"
type={this.state.secretKeyVisible ? "text" : "password"}
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
readonly={this.state.keysReadOnly}
/>
<i
onClick={() => {
this.setState({
newSecretKeyVisible: !this.state.newSecretKeyVisible
})
}}
className={
"toggle-password fa fa-eye " +
(this.state.newSecretKeyVisible ? "toggled" : "")
}
/>
<InputGroup
value={this.state.newSecretKey}
onChange={e => {
this.setState({ newSecretKey: e.target.value })
}}
id="newSecretKey"
label="New Secret Key"
name="newSecretKey"
type={this.state.newSecretKeyVisible ? "text" : "password"}
spellCheck="false"
required="required"
autoComplete="false"
align="ig-left"
onChange={e => {
this.setState({ newSecretKey: e.target.value })
}}
/>
</div>
</ModalBody>
<div className="modal-footer">
<button
id="generate-keys"
className={
"btn btn-primary " + (this.state.keysReadOnly ? "hidden" : "")
}
className={"btn btn-primary"}
onClick={this.generateAuth.bind(this)}
>
Generate
</button>
<button
id="update-keys"
className={
"btn btn-success " + (this.state.keysReadOnly ? "hidden" : "")
}
className={classNames({
btn: true,
"btn-success": this.canUpdateCredentials()
})}
disabled={!this.canUpdateCredentials()}
onClick={this.setAuth.bind(this)}
>
Update
@@ -198,4 +278,7 @@ const mapDispatchToProps = dispatch => {
}
}
export default connect(mapStateToProps, mapDispatchToProps)(ChangePasswordModal)
export default connect(
mapStateToProps,
mapDispatchToProps
)(ChangePasswordModal)
+8 -2
View File
@@ -48,7 +48,7 @@ export class Login extends React.Component {
handleSubmit(event) {
event.preventDefault()
const { showAlert, history } = this.props
const { showAlert, clearAlert, history } = this.props
let message = ""
if (this.state.accessKey === "") {
message = "Access Key cannot be empty"
@@ -66,6 +66,9 @@ export class Login extends React.Component {
password: this.state.secretKey
})
.then(res => {
// Clear alerts from previous login attempts
clearAlert()
history.push("/")
})
.catch(e => {
@@ -145,4 +148,7 @@ const mapDispatchToProps = dispatch => {
}
}
export default connect(state => state, mapDispatchToProps)(Login)
export default connect(
state => state,
mapDispatchToProps
)(Login)
@@ -17,21 +17,38 @@
import React from "react"
import { shallow, mount } from "enzyme"
import { ChangePasswordModal } from "../ChangePasswordModal"
import jwtDecode from "jwt-decode"
jest.mock("jwt-decode")
jwtDecode.mockImplementation(() => ({ sub: "minio" }))
jest.mock("../../web", () => ({
GetAuth: jest.fn(() => {
return Promise.resolve({ accessKey: "test1", secretKey: "test2" })
}),
GenerateAuth: jest.fn(() => {
return Promise.resolve({ accessKey: "gen1", secretKey: "gen2" })
}),
SetAuth: jest.fn(({ accessKey, secretKey }) => {
if (accessKey == "test3" && secretKey == "test4") {
return Promise.resolve({})
} else {
return Promise.reject({ message: "Error" })
SetAuth: jest.fn(
({ currentAccessKey, currentSecretKey, newAccessKey, newSecretKey }) => {
if (
currentAccessKey == "minio" &&
currentSecretKey == "minio123" &&
newAccessKey == "test" &&
newSecretKey == "test1234"
) {
return Promise.resolve({})
} else {
return Promise.reject({
message: "Error"
})
}
}
})
),
GetToken: jest.fn(() => "")
}))
jest.mock("../../utils", () => ({
getRandomAccessKey: () => "raccesskey",
getRandomSecretKey: () => "rsecretkey"
}))
describe("ChangePasswordModal", () => {
@@ -40,57 +57,112 @@ describe("ChangePasswordModal", () => {
memory: "test",
platform: "test",
runtime: "test",
info: { isEnvCreds: false }
info: { isEnvCreds: false },
userInfo: { isIAMUser: false }
}
it("should render without crashing", () => {
shallow(<ChangePasswordModal serverInfo={serverInfo} />)
})
it("should get the keys when its rendered", () => {
const wrapper = shallow(<ChangePasswordModal serverInfo={serverInfo} />)
setImmediate(() => {
expect(wrapper.state("accessKey")).toBe("test1")
expect(wrapper.state("secretKey")).toBe("test2")
})
it("should not allow changing password when isWorm is true", () => {
const newServerInfo = { ...serverInfo, info: { isWorm: true } }
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
expect(
wrapper
.find("ModalBody")
.childAt(0)
.text()
).toBe("Credentials of this user cannot be updated through MinIO Browser.")
})
it("should show readonly keys when isEnvCreds is true", () => {
const newServerInfo = { ...serverInfo, info: { isEnvCreds: true } }
it("should not allow changing password when isEnvCreds is true and not IAM user", () => {
const newServerInfo = {
...serverInfo,
info: { isEnvCreds: true },
userInfo: { isIAMUser: false }
}
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
expect(wrapper.state("accessKey")).toBe("xxxxxxxxx")
expect(wrapper.state("secretKey")).toBe("xxxxxxxxx")
expect(wrapper.find("#accessKey").prop("readonly")).toBeTruthy()
expect(wrapper.find("#secretKey").prop("readonly")).toBeTruthy()
expect(wrapper.find("#generate-keys").hasClass("hidden")).toBeTruthy()
expect(wrapper.find("#update-keys").hasClass("hidden")).toBeTruthy()
expect(
wrapper
.find("ModalBody")
.childAt(0)
.text()
).toBe("Credentials of this user cannot be updated through MinIO Browser.")
})
it("should generate accessKey and secretKey when Generate buttons is clicked", () => {
const wrapper = shallow(<ChangePasswordModal serverInfo={serverInfo} />)
wrapper.find("#generate-keys").simulate("click")
setImmediate(() => {
expect(wrapper.state("accessKey")).toBe("gen1")
expect(wrapper.state("secretKey")).toBe("gen2")
expect(wrapper.state("newAccessKey")).toBe("raccesskey")
expect(wrapper.state("newSecretKey")).toBe("rsecretkey")
})
})
it("should not generate accessKey for IAM User", () => {
const newServerInfo = {
...serverInfo,
userInfo: { isIAMUser: true }
}
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
wrapper.find("#generate-keys").simulate("click")
setImmediate(() => {
expect(wrapper.state("newAccessKey")).toBe("minio")
expect(wrapper.state("newSecretKey")).toBe("rsecretkey")
})
})
it("should not show new accessKey field for IAM User", () => {
const newServerInfo = {
...serverInfo,
userInfo: { isIAMUser: true }
}
const wrapper = shallow(<ChangePasswordModal serverInfo={newServerInfo} />)
expect(wrapper.find("#newAccesskey").exists()).toBeFalsy()
})
it("should disble Update button for invalid accessKey or secretKey", () => {
const showAlert = jest.fn()
const wrapper = shallow(
<ChangePasswordModal serverInfo={serverInfo} showAlert={showAlert} />
)
wrapper
.find("#currentAccessKey")
.simulate("change", { target: { value: "minio" } })
wrapper
.find("#currentSecretKey")
.simulate("change", { target: { value: "minio123" } })
wrapper.find("#newAccessKey").simulate("change", { target: { value: "t" } })
wrapper
.find("#newSecretKey")
.simulate("change", { target: { value: "t1" } })
expect(wrapper.find("#update-keys").prop("disabled")).toBeTruthy()
})
it("should update accessKey and secretKey when Update button is clicked", () => {
const showAlert = jest.fn()
const wrapper = shallow(
<ChangePasswordModal serverInfo={serverInfo} showAlert={showAlert} />
)
wrapper
.find("#accessKey")
.simulate("change", { target: { value: "test3" } })
.find("#currentAccessKey")
.simulate("change", { target: { value: "minio" } })
wrapper
.find("#secretKey")
.simulate("change", { target: { value: "test4" } })
.find("#currentSecretKey")
.simulate("change", { target: { value: "minio123" } })
wrapper
.find("#newAccessKey")
.simulate("change", { target: { value: "test" } })
wrapper
.find("#newSecretKey")
.simulate("change", { target: { value: "test1234" } })
expect(wrapper.find("#update-keys").prop("disabled")).toBeFalsy()
wrapper.find("#update-keys").simulate("click")
setImmediate(() => {
expect(showAlert).toHaveBeenCalledWith({
type: "success",
message: "Changed credentials"
message: "Credentials updated successfully."
})
})
})
+3 -2
View File
@@ -34,7 +34,7 @@ export const fetchStorageInfo = () => {
return web.StorageInfo().then(res => {
const storageInfo = {
total: res.storageInfo.Total,
used: res.storageInfo.Used
used: res.storageInfo.Used
}
dispatch(setStorageInfo(storageInfo))
})
@@ -54,7 +54,8 @@ export const fetchServerInfo = () => {
memory: res.MinioMemory,
platform: res.MinioPlatform,
runtime: res.MinioRuntime,
info: res.MinioGlobalInfo
info: res.MinioGlobalInfo,
userInfo: res.MinioUserInfo
}
dispatch(setServerInfo(serverInfo))
})
+10
View File
@@ -28,3 +28,13 @@ export const NONE = "none"
export const SHARE_OBJECT_EXPIRY_DAYS = 5
export const SHARE_OBJECT_EXPIRY_HOURS = 0
export const SHARE_OBJECT_EXPIRY_MINUTES = 0
export const ACCESS_KEY_MIN_LENGTH = 3
export const SECRET_KEY_MIN_LENGTH = 8
export const SORT_BY_NAME = "name"
export const SORT_BY_SIZE = "size"
export const SORT_BY_LAST_MODIFIED = "last-modified"
export const SORT_ORDER_ASC = "asc"
export const SORT_ORDER_DESC = "desc"
+1 -1
View File
@@ -51,7 +51,7 @@ export const ObjectItem = ({
</div>
<div className="fesl-item fesl-item-name">
<a
href="#"
href={getDataType(name, contentType) === "folder" ? name : "#"}
onClick={e => {
e.preventDefault()
if (onClick) {
+34 -17
View File
@@ -18,11 +18,19 @@ import React from "react"
import classNames from "classnames"
import { connect } from "react-redux"
import * as actionsObjects from "./actions"
import {
SORT_BY_NAME,
SORT_BY_SIZE,
SORT_BY_LAST_MODIFIED,
SORT_ORDER_DESC,
SORT_ORDER_ASC
} from "../constants"
export const ObjectsHeader = ({
sortNameOrder,
sortSizeOrder,
sortLastModifiedOrder,
sortedByName,
sortedBySize,
sortedByLastModified,
sortOrder,
sortObjects
}) => (
<div className="feb-container">
@@ -31,48 +39,54 @@ export const ObjectsHeader = ({
<div
className="fesl-item fesl-item-name"
id="sort-by-name"
onClick={() => sortObjects("name")}
onClick={() => sortObjects(SORT_BY_NAME)}
data-sort="name"
>
Name
<i
className={classNames({
"fesli-sort": true,
"fesli-sort--active": sortedByName,
fa: true,
"fa-sort-alpha-desc": sortNameOrder,
"fa-sort-alpha-asc": !sortNameOrder
"fa-sort-alpha-desc": sortedByName && sortOrder === SORT_ORDER_DESC,
"fa-sort-alpha-asc": sortedByName && sortOrder === SORT_ORDER_ASC
})}
/>
</div>
<div
className="fesl-item fesl-item-size"
id="sort-by-size"
onClick={() => sortObjects("size")}
onClick={() => sortObjects(SORT_BY_SIZE)}
data-sort="size"
>
Size
<i
className={classNames({
"fesli-sort": true,
"fesli-sort--active": sortedBySize,
fa: true,
"fa-sort-amount-desc": sortSizeOrder,
"fa-sort-amount-asc": !sortSizeOrder
"fa-sort-amount-desc":
sortedBySize && sortOrder === SORT_ORDER_DESC,
"fa-sort-amount-asc": sortedBySize && sortOrder === SORT_ORDER_ASC
})}
/>
</div>
<div
className="fesl-item fesl-item-modified"
id="sort-by-last-modified"
onClick={() => sortObjects("last-modified")}
onClick={() => sortObjects(SORT_BY_LAST_MODIFIED)}
data-sort="last-modified"
>
Last Modified
<i
className={classNames({
"fesli-sort": true,
"fesli-sort--active": sortedByLastModified,
fa: true,
"fa-sort-numeric-desc": sortLastModifiedOrder,
"fa-sort-numeric-asc": !sortLastModifiedOrder
"fa-sort-numeric-desc":
sortedByLastModified && sortOrder === SORT_ORDER_DESC,
"fa-sort-numeric-asc":
sortedByLastModified && sortOrder === SORT_ORDER_ASC
})}
/>
</div>
@@ -83,10 +97,10 @@ export const ObjectsHeader = ({
const mapStateToProps = state => {
return {
sortNameOrder: state.objects.sortBy == "name" && state.objects.sortOrder,
sortSizeOrder: state.objects.sortBy == "size" && state.objects.sortOrder,
sortLastModifiedOrder:
state.objects.sortBy == "last-modified" && state.objects.sortOrder
sortedByName: state.objects.sortBy == SORT_BY_NAME,
sortedBySize: state.objects.sortBy == SORT_BY_SIZE,
sortedByLastModified: state.objects.sortBy == SORT_BY_LAST_MODIFIED,
sortOrder: state.objects.sortOrder
}
}
@@ -96,4 +110,7 @@ const mapDispatchToProps = dispatch => {
}
}
export default connect(mapStateToProps, mapDispatchToProps)(ObjectsHeader)
export default connect(
mapStateToProps,
mapDispatchToProps
)(ObjectsHeader)
+37 -23
View File
@@ -15,32 +15,52 @@
*/
import React from "react"
import classNames from "classnames"
import { connect } from "react-redux"
import InfiniteScroll from "react-infinite-scroller"
import * as actionsObjects from "./actions"
import ObjectsList from "./ObjectsList"
export class ObjectsListContainer extends React.Component {
constructor(props) {
super(props)
this.state = {
page: 1
}
this.loadNextPage = this.loadNextPage.bind(this)
}
componentWillReceiveProps(nextProps) {
if (
nextProps.currentBucket !== this.props.currentBucket ||
nextProps.currentPrefix !== this.props.currentPrefix ||
nextProps.sortBy !== this.props.sortBy ||
nextProps.sortOrder !== this.props.sortOrder
) {
this.setState({
page: 1
})
}
}
loadNextPage() {
this.setState(state => {
return { page: state.page + 1 }
})
}
render() {
const { objects, isTruncated, currentBucket, loadObjects } = this.props
const { objects, listLoading } = this.props
const visibleObjects = objects.slice(0, this.state.page * 100)
return (
<div className="feb-container">
<div style={{ position: "relative" }}>
<InfiniteScroll
pageStart={0}
loadMore={() => loadObjects(true)}
hasMore={isTruncated}
loadMore={this.loadNextPage}
hasMore={objects.length > visibleObjects.length}
useWindow={true}
initialLoad={false}
>
<ObjectsList objects={objects} />
<ObjectsList objects={visibleObjects} />
</InfiniteScroll>
<div
className="text-center"
style={{ display: isTruncated && currentBucket ? "block" : "none" }}
>
<span>Loading...</span>
</div>
{listLoading && <div className="loading" />}
</div>
)
}
@@ -51,16 +71,10 @@ const mapStateToProps = state => {
currentBucket: state.buckets.currentBucket,
currentPrefix: state.objects.currentPrefix,
objects: state.objects.list,
isTruncated: state.objects.isTruncated
sortBy: state.objects.sortBy,
sortOrder: state.objects.sortOrder,
listLoading: state.objects.listLoading
}
}
const mapDispatchToProps = dispatch => {
return {
loadObjects: append => dispatch(actionsObjects.fetchObjects(append))
}
}
export default connect(mapStateToProps, mapDispatchToProps)(
ObjectsListContainer
)
export default connect(mapStateToProps)(ObjectsListContainer)
@@ -17,6 +17,7 @@
import React from "react"
import { shallow } from "enzyme"
import { ObjectsHeader } from "../ObjectsHeader"
import { SORT_ORDER_ASC, SORT_ORDER_DESC } from "../../constants"
describe("ObjectsHeader", () => {
it("should render without crashing", () => {
@@ -24,44 +25,84 @@ describe("ObjectsHeader", () => {
shallow(<ObjectsHeader sortObjects={sortObjects} />)
})
it("should render columns with asc classes by default", () => {
it("should render the name column with asc class when objects are sorted by name asc", () => {
const sortObjects = jest.fn()
const wrapper = shallow(<ObjectsHeader sortObjects={sortObjects} />)
const wrapper = shallow(
<ObjectsHeader
sortObjects={sortObjects}
sortedByName={true}
sortOrder={SORT_ORDER_ASC}
/>
)
expect(
wrapper.find("#sort-by-name i").hasClass("fa-sort-alpha-asc")
).toBeTruthy()
expect(
wrapper.find("#sort-by-size i").hasClass("fa-sort-amount-asc")
).toBeTruthy()
expect(
wrapper.find("#sort-by-last-modified i").hasClass("fa-sort-numeric-asc")
).toBeTruthy()
})
it("should render name column with desc class when objects are sorted by name", () => {
it("should render the name column with desc class when objects are sorted by name desc", () => {
const sortObjects = jest.fn()
const wrapper = shallow(
<ObjectsHeader sortObjects={sortObjects} sortNameOrder={true} />
<ObjectsHeader
sortObjects={sortObjects}
sortedByName={true}
sortOrder={SORT_ORDER_DESC}
/>
)
expect(
wrapper.find("#sort-by-name i").hasClass("fa-sort-alpha-desc")
).toBeTruthy()
})
it("should render size column with desc class when objects are sorted by size", () => {
it("should render the size column with asc class when objects are sorted by size asc", () => {
const sortObjects = jest.fn()
const wrapper = shallow(
<ObjectsHeader sortObjects={sortObjects} sortSizeOrder={true} />
<ObjectsHeader
sortObjects={sortObjects}
sortedBySize={true}
sortOrder={SORT_ORDER_ASC}
/>
)
expect(
wrapper.find("#sort-by-size i").hasClass("fa-sort-amount-asc")
).toBeTruthy()
})
it("should render the size column with desc class when objects are sorted by size desc", () => {
const sortObjects = jest.fn()
const wrapper = shallow(
<ObjectsHeader
sortObjects={sortObjects}
sortedBySize={true}
sortOrder={SORT_ORDER_DESC}
/>
)
expect(
wrapper.find("#sort-by-size i").hasClass("fa-sort-amount-desc")
).toBeTruthy()
})
it("should render last modified column with desc class when objects are sorted by last modified", () => {
it("should render the date column with asc class when objects are sorted by date asc", () => {
const sortObjects = jest.fn()
const wrapper = shallow(
<ObjectsHeader sortObjects={sortObjects} sortLastModifiedOrder={true} />
<ObjectsHeader
sortObjects={sortObjects}
sortedByLastModified={true}
sortOrder={SORT_ORDER_ASC}
/>
)
expect(
wrapper.find("#sort-by-last-modified i").hasClass("fa-sort-numeric-asc")
).toBeTruthy()
})
it("should render the date column with desc class when objects are sorted by date desc", () => {
const sortObjects = jest.fn()
const wrapper = shallow(
<ObjectsHeader
sortObjects={sortObjects}
sortedByLastModified={true}
sortOrder={SORT_ORDER_DESC}
/>
)
expect(
wrapper.find("#sort-by-last-modified i").hasClass("fa-sort-numeric-desc")
@@ -20,14 +20,13 @@ import { ObjectsListContainer } from "../ObjectsListContainer"
describe("ObjectsList", () => {
it("should render without crashing", () => {
shallow(<ObjectsListContainer loadObjects={jest.fn()} />)
shallow(<ObjectsListContainer objects={[]} />)
})
it("should render ObjectsList with objects", () => {
const wrapper = shallow(
<ObjectsListContainer
objects={[{ name: "test1.jpg" }, { name: "test2.jpg" }]}
loadObjects={jest.fn()}
/>
)
expect(wrapper.find("ObjectsList").length).toBe(1)
@@ -37,10 +36,14 @@ describe("ObjectsList", () => {
])
})
it("should show the loading indicator at the bottom if there are more elements to display", () => {
it("should show the loading indicator when the objects are being loaded", () => {
const wrapper = shallow(
<ObjectsListContainer currentBucket="test1" isTruncated={true} />
<ObjectsListContainer
currentBucket="test1"
objects={[]}
listLoading={true}
/>
)
expect(wrapper.find(".text-center").prop("style")).toHaveProperty("display", "block")
expect(wrapper.find(".loading").exists()).toBeTruthy()
})
})
@@ -18,7 +18,13 @@ import configureStore from "redux-mock-store"
import thunk from "redux-thunk"
import * as actionsObjects from "../actions"
import * as alertActions from "../../alert/actions"
import { minioBrowserPrefix } from "../../constants"
import {
minioBrowserPrefix,
SORT_BY_NAME,
SORT_ORDER_ASC,
SORT_BY_LAST_MODIFIED,
SORT_ORDER_DESC
} from "../../constants"
import history from "../../history"
jest.mock("../../web", () => ({
@@ -37,8 +43,6 @@ jest.mock("../../web", () => ({
} else {
return Promise.resolve({
objects: [{ name: "test1" }, { name: "test2" }],
istruncated: false,
nextmarker: "test2",
writable: false
})
}
@@ -77,17 +81,11 @@ describe("Objects actions", () => {
const expectedActions = [
{
type: "objects/SET_LIST",
objects: [{ name: "test1" }, { name: "test2" }],
isTruncated: false,
marker: "test2"
objects: [{ name: "test1" }, { name: "test2" }]
}
]
store.dispatch(
actionsObjects.setList(
[{ name: "test1" }, { name: "test2" }],
"test2",
false
)
actionsObjects.setList([{ name: "test1" }, { name: "test2" }])
)
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
@@ -98,10 +96,10 @@ describe("Objects actions", () => {
const expectedActions = [
{
type: "objects/SET_SORT_BY",
sortBy: "name"
sortBy: SORT_BY_NAME
}
]
store.dispatch(actionsObjects.setSortBy("name"))
store.dispatch(actionsObjects.setSortBy(SORT_BY_NAME))
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
@@ -111,10 +109,10 @@ describe("Objects actions", () => {
const expectedActions = [
{
type: "objects/SET_SORT_ORDER",
sortOrder: true
sortOrder: SORT_ORDER_ASC
}
]
store.dispatch(actionsObjects.setSortOrder(true))
store.dispatch(actionsObjects.setSortOrder(SORT_ORDER_ASC))
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
@@ -126,23 +124,26 @@ describe("Objects actions", () => {
})
const expectedActions = [
{
type: "objects/SET_LIST",
objects: [{ name: "test1" }, { name: "test2" }],
marker: "test2",
isTruncated: false
type: "objects/RESET_LIST"
},
{ listLoading: true, type: "objects/SET_LIST_LOADING" },
{
type: "objects/SET_SORT_BY",
sortBy: ""
sortBy: SORT_BY_LAST_MODIFIED
},
{
type: "objects/SET_SORT_ORDER",
sortOrder: false
sortOrder: SORT_ORDER_DESC
},
{
type: "objects/SET_LIST",
objects: [{ name: "test2" }, { name: "test1" }]
},
{
type: "objects/SET_PREFIX_WRITABLE",
prefixWritable: false
}
},
{ listLoading: false, type: "objects/SET_LIST_LOADING" }
]
return store.dispatch(actionsObjects.fetchObjects()).then(() => {
const actions = store.getActions()
@@ -150,35 +151,16 @@ describe("Objects actions", () => {
})
})
it("creates objects/APPEND_LIST after fetching more objects", () => {
const store = mockStore({
buckets: { currentBucket: "bk1" },
objects: { currentPrefix: "" }
})
const expectedActions = [
{
type: "objects/APPEND_LIST",
objects: [{ name: "test1" }, { name: "test2" }],
marker: "test2",
isTruncated: false
},
{
type: "objects/SET_PREFIX_WRITABLE",
prefixWritable: false
}
]
return store.dispatch(actionsObjects.fetchObjects(true)).then(() => {
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
})
it("creates objects/RESET_LIST after failing to fetch the objects from bucket with ListObjects denied for LoggedIn users", () => {
const store = mockStore({
buckets: { currentBucket: "test-deny" },
objects: { currentPrefix: "" }
})
const expectedActions = [
{
type: "objects/RESET_LIST"
},
{ listLoading: true, type: "objects/SET_LIST_LOADING" },
{
type: "alert/SET",
alert: {
@@ -189,8 +171,9 @@ describe("Objects actions", () => {
}
},
{
type: "object/RESET_LIST"
}
type: "objects/RESET_LIST"
},
{ listLoading: false, type: "objects/SET_LIST_LOADING" }
]
return store.dispatch(actionsObjects.fetchObjects()).then(() => {
const actions = store.getActions()
@@ -213,28 +196,24 @@ describe("Objects actions", () => {
objects: {
list: [],
sortBy: "",
sortOrder: false,
isTruncated: false,
marker: ""
sortOrder: SORT_ORDER_ASC
}
})
const expectedActions = [
{
type: "objects/SET_SORT_BY",
sortBy: "name"
sortBy: SORT_BY_NAME
},
{
type: "objects/SET_SORT_ORDER",
sortOrder: true
sortOrder: SORT_ORDER_ASC
},
{
type: "objects/SET_LIST",
objects: [],
isTruncated: false,
marker: ""
objects: []
}
]
store.dispatch(actionsObjects.sortObjects("name"))
store.dispatch(actionsObjects.sortObjects(SORT_BY_NAME))
const actions = store.getActions()
expect(actions).toEqual(expectedActions)
})
@@ -246,6 +225,10 @@ describe("Objects actions", () => {
})
const expectedActions = [
{ type: "objects/SET_CURRENT_PREFIX", prefix: "abc/" },
{
type: "objects/RESET_LIST"
},
{ listLoading: true, type: "objects/SET_LIST_LOADING" },
{ type: "objects/CHECKED_LIST_RESET" }
]
store.dispatch(actionsObjects.selectPrefix("abc/"))
@@ -16,17 +16,17 @@
import reducer from "../reducer"
import * as actions from "../actions"
import { SORT_ORDER_ASC, SORT_BY_NAME } from "../../constants"
describe("objects reducer", () => {
it("should return the initial state", () => {
const initialState = reducer(undefined, {})
expect(initialState).toEqual({
list: [],
listLoading: false,
sortBy: "",
sortOrder: false,
sortOrder: SORT_ORDER_ASC,
currentPrefix: "",
marker: "",
isTruncated: false,
prefixWritable: false,
shareObject: {
show: false,
@@ -40,37 +40,9 @@ describe("objects reducer", () => {
it("should handle SET_LIST", () => {
const newState = reducer(undefined, {
type: actions.SET_LIST,
objects: [{ name: "obj1" }, { name: "obj2" }],
marker: "obj2",
isTruncated: false
objects: [{ name: "obj1" }, { name: "obj2" }]
})
expect(newState.list).toEqual([{ name: "obj1" }, { name: "obj2" }])
expect(newState.marker).toBe("obj2")
expect(newState.isTruncated).toBeFalsy()
})
it("should handle APPEND_LIST", () => {
const newState = reducer(
{
list: [{ name: "obj1" }, { name: "obj2" }],
marker: "obj2",
isTruncated: true
},
{
type: actions.APPEND_LIST,
objects: [{ name: "obj3" }, { name: "obj4" }],
marker: "obj4",
isTruncated: false
}
)
expect(newState.list).toEqual([
{ name: "obj1" },
{ name: "obj2" },
{ name: "obj3" },
{ name: "obj4" }
])
expect(newState.marker).toBe("obj4")
expect(newState.isTruncated).toBeFalsy()
})
it("should handle REMOVE", () => {
@@ -98,30 +70,28 @@ describe("objects reducer", () => {
it("should handle SET_SORT_BY", () => {
const newState = reducer(undefined, {
type: actions.SET_SORT_BY,
sortBy: "name"
sortBy: SORT_BY_NAME
})
expect(newState.sortBy).toEqual("name")
expect(newState.sortBy).toEqual(SORT_BY_NAME)
})
it("should handle SET_SORT_ORDER", () => {
const newState = reducer(undefined, {
type: actions.SET_SORT_ORDER,
sortOrder: true
sortOrder: SORT_ORDER_ASC
})
expect(newState.sortOrder).toEqual(true)
expect(newState.sortOrder).toEqual(SORT_ORDER_ASC)
})
it("should handle SET_CURRENT_PREFIX", () => {
const newState = reducer(
{ currentPrefix: "test1/", marker: "abc", isTruncated: true },
{ currentPrefix: "test1/" },
{
type: actions.SET_CURRENT_PREFIX,
prefix: "test2/"
}
)
expect(newState.currentPrefix).toEqual("test2/")
expect(newState.marker).toEqual("")
expect(newState.isTruncated).toBeFalsy()
})
it("should handle SET_PREFIX_WRITABLE", () => {
+94 -59
View File
@@ -16,15 +16,26 @@
import web from "../web"
import history from "../history"
import { sortObjectsByName, sortObjectsBySize, sortObjectsByDate } from "../utils"
import {
sortObjectsByName,
sortObjectsBySize,
sortObjectsByDate
} from "../utils"
import { getCurrentBucket } from "../buckets/selectors"
import { getCurrentPrefix, getCheckedList } from "./selectors"
import * as alertActions from "../alert/actions"
import * as bucketActions from "../buckets/actions"
import { minioBrowserPrefix } from "../constants"
import {
minioBrowserPrefix,
SORT_BY_NAME,
SORT_BY_SIZE,
SORT_BY_LAST_MODIFIED,
SORT_ORDER_ASC,
SORT_ORDER_DESC
} from "../constants"
export const SET_LIST = "objects/SET_LIST"
export const RESET_LIST = "object/RESET_LIST"
export const RESET_LIST = "objects/RESET_LIST"
export const APPEND_LIST = "objects/APPEND_LIST"
export const REMOVE = "objects/REMOVE"
export const SET_SORT_BY = "objects/SET_SORT_BY"
@@ -35,53 +46,63 @@ export const SET_SHARE_OBJECT = "objects/SET_SHARE_OBJECT"
export const CHECKED_LIST_ADD = "objects/CHECKED_LIST_ADD"
export const CHECKED_LIST_REMOVE = "objects/CHECKED_LIST_REMOVE"
export const CHECKED_LIST_RESET = "objects/CHECKED_LIST_RESET"
export const SET_LIST_LOADING = "objects/SET_LIST_LOADING"
export const setList = (objects, marker, isTruncated) => ({
export const setList = objects => ({
type: SET_LIST,
objects,
marker,
isTruncated
objects
})
export const resetList = () => ({
type: RESET_LIST
})
export const appendList = (objects, marker, isTruncated) => ({
type: APPEND_LIST,
objects,
marker,
isTruncated
export const setListLoading = listLoading => ({
type: SET_LIST_LOADING,
listLoading
})
export const fetchObjects = append => {
export const fetchObjects = () => {
return function(dispatch, getState) {
const {buckets: {currentBucket}, objects: {currentPrefix, marker}} = getState()
dispatch(resetList())
const {
buckets: { currentBucket },
objects: { currentPrefix }
} = getState()
if (currentBucket) {
dispatch(setListLoading(true))
return web
.ListObjects({
bucketName: currentBucket,
prefix: currentPrefix,
marker: append ? marker : ""
prefix: currentPrefix
})
.then(res => {
let objects = []
if (res.objects) {
objects = res.objects.map(object => {
return {
...object,
name: object.name.replace(currentPrefix, "")
}
})
// we need to check if the bucket name and prefix are the same as
// when the request was made before updating the displayed objects
if (
currentBucket === getCurrentBucket(getState()) &&
currentPrefix === getCurrentPrefix(getState())
) {
let objects = []
if (res.objects) {
objects = res.objects.map(object => {
return {
...object,
name: object.name.replace(currentPrefix, "")
}
})
}
const sortBy = SORT_BY_LAST_MODIFIED
const sortOrder = SORT_ORDER_DESC
dispatch(setSortBy(sortBy))
dispatch(setSortOrder(sortOrder))
const sortedList = sortObjectsList(objects, sortBy, sortOrder)
dispatch(setList(sortedList))
dispatch(setPrefixWritable(res.writable))
dispatch(setListLoading(false))
}
if (append) {
dispatch(appendList(objects, res.nextmarker, res.istruncated))
} else {
dispatch(setList(objects, res.nextmarker, res.istruncated))
dispatch(setSortBy(""))
dispatch(setSortOrder(false))
}
dispatch(setPrefixWritable(res.writable))
})
.catch(err => {
if (web.LoggedIn()) {
@@ -96,6 +117,7 @@ export const fetchObjects = append => {
} else {
history.push("/login")
}
dispatch(setListLoading(false))
})
}
}
@@ -103,26 +125,27 @@ export const fetchObjects = append => {
export const sortObjects = sortBy => {
return function(dispatch, getState) {
const {objects} = getState()
const sortOrder = objects.sortBy == sortBy ? !objects.sortOrder : true
const { objects } = getState()
let sortOrder = SORT_ORDER_ASC
// Reverse sort order if the list is already sorted on same field
if (objects.sortBy === sortBy && objects.sortOrder === SORT_ORDER_ASC) {
sortOrder = SORT_ORDER_DESC
}
dispatch(setSortBy(sortBy))
dispatch(setSortOrder(sortOrder))
let list
switch (sortBy) {
case "name":
list = sortObjectsByName(objects.list, sortOrder)
break
case "size":
list = sortObjectsBySize(objects.list, sortOrder)
break
case "last-modified":
list = sortObjectsByDate(objects.list, sortOrder)
break
default:
list = objects.list
break
}
dispatch(setList(list, objects.marker, objects.isTruncated))
const sortedList = sortObjectsList(objects.list, sortBy, sortOrder)
dispatch(setList(sortedList))
}
}
const sortObjectsList = (list, sortBy, sortOrder) => {
switch (sortBy) {
case SORT_BY_NAME:
return sortObjectsByName(list, sortOrder)
case SORT_BY_SIZE:
return sortObjectsBySize(list, sortOrder)
case SORT_BY_LAST_MODIFIED:
return sortObjectsByDate(list, sortOrder)
}
}
@@ -208,7 +231,8 @@ export const shareObject = (object, days, hours, minutes) => {
.PresignedGet({
host: location.host,
bucket: currentBucket,
object: objectName
object: objectName,
expiry: expiry
})
.then(obj => {
dispatch(showShareObject(object, obj.url))
@@ -228,7 +252,16 @@ export const shareObject = (object, days, hours, minutes) => {
)
})
} else {
dispatch(showShareObject(object, `${location.host}` + '/' + `${currentBucket}` + '/' + encodeURI(objectName)))
dispatch(
showShareObject(
object,
`${location.host}` +
"/" +
`${currentBucket}` +
"/" +
encodeURI(objectName)
)
)
dispatch(
alertActions.set({
type: "success",
@@ -321,13 +354,14 @@ export const downloadCheckedObjects = () => {
}${minioBrowserPrefix}/zip?token=${res.token}`
downloadZip(requestUrl, req, dispatch)
})
.catch(err => dispatch(
alertActions.set({
type: "danger",
message: err.message
})
.catch(err =>
dispatch(
alertActions.set({
type: "danger",
message: err.message
})
)
)
)
}
}
}
@@ -350,7 +384,8 @@ const downloadZip = (url, req, dispatch) => {
var separator = req.prefix.length > 1 ? "-" : ""
anchor.href = blobUrl
anchor.download = req.bucketName + separator + req.prefix.slice(0, -1) + ".zip"
anchor.download =
req.bucketName + separator + req.prefix.slice(0, -1) + ".zip"
anchor.click()
window.URL.revokeObjectURL(blobUrl)
+8 -16
View File
@@ -15,6 +15,7 @@
*/
import * as actionsObjects from "./actions"
import { SORT_ORDER_ASC } from "../constants"
const removeObject = (list, objectToRemove, lookup) => {
const idx = list.findIndex(object => lookup(object) === objectToRemove)
@@ -27,11 +28,10 @@ const removeObject = (list, objectToRemove, lookup) => {
export default (
state = {
list: [],
listLoading: false,
sortBy: "",
sortOrder: false,
sortOrder: SORT_ORDER_ASC,
currentPrefix: "",
marker: "",
isTruncated: false,
prefixWritable: false,
shareObject: {
show: false,
@@ -46,23 +46,17 @@ export default (
case actionsObjects.SET_LIST:
return {
...state,
list: action.objects,
marker: action.marker,
isTruncated: action.isTruncated
list: action.objects
}
case actionsObjects.RESET_LIST:
return {
...state,
list: [],
marker: "",
isTruncated: false
list: []
}
case actionsObjects.APPEND_LIST:
case actionsObjects.SET_LIST_LOADING:
return {
...state,
list: [...state.list, ...action.objects],
marker: action.marker,
isTruncated: action.isTruncated
listLoading: action.listLoading
}
case actionsObjects.REMOVE:
return {
@@ -82,9 +76,7 @@ export default (
case actionsObjects.SET_CURRENT_PREFIX:
return {
...state,
currentPrefix: action.prefix,
marker: "",
isTruncated: false
currentPrefix: action.prefix
}
case actionsObjects.SET_PREFIX_WRITABLE:
return {
+47 -23
View File
@@ -14,11 +14,11 @@
* limitations under the License.
*/
import { minioBrowserPrefix } from './constants.js'
import { minioBrowserPrefix, SORT_ORDER_DESC } from "./constants.js"
export const sortObjectsByName = (objects, order) => {
let folders = objects.filter(object => object.name.endsWith('/'))
let files = objects.filter(object => !object.name.endsWith('/'))
let folders = objects.filter(object => object.name.endsWith("/"))
let files = objects.filter(object => !object.name.endsWith("/"))
folders = folders.sort((a, b) => {
if (a.name.toLowerCase() < b.name.toLowerCase()) return -1
if (a.name.toLowerCase() > b.name.toLowerCase()) return 1
@@ -29,7 +29,7 @@ export const sortObjectsByName = (objects, order) => {
if (a.name.toLowerCase() > b.name.toLowerCase()) return 1
return 0
})
if (order) {
if (order === SORT_ORDER_DESC) {
folders = folders.reverse()
files = files.reverse()
}
@@ -37,32 +37,34 @@ export const sortObjectsByName = (objects, order) => {
}
export const sortObjectsBySize = (objects, order) => {
let folders = objects.filter(object => object.name.endsWith('/'))
let files = objects.filter(object => !object.name.endsWith('/'))
let folders = objects.filter(object => object.name.endsWith("/"))
let files = objects.filter(object => !object.name.endsWith("/"))
files = files.sort((a, b) => a.size - b.size)
if (order)
files = files.reverse()
if (order === SORT_ORDER_DESC) files = files.reverse()
return [...folders, ...files]
}
export const sortObjectsByDate = (objects, order) => {
let folders = objects.filter(object => object.name.endsWith('/'))
let files = objects.filter(object => !object.name.endsWith('/'))
files = files.sort((a, b) => new Date(a.lastModified).getTime() - new Date(b.lastModified).getTime())
if (order)
files = files.reverse()
let folders = objects.filter(object => object.name.endsWith("/"))
let files = objects.filter(object => !object.name.endsWith("/"))
files = files.sort(
(a, b) =>
new Date(a.lastModified).getTime() - new Date(b.lastModified).getTime()
)
if (order === SORT_ORDER_DESC) files = files.reverse()
return [...folders, ...files]
}
export const pathSlice = (path) => {
path = path.replace(minioBrowserPrefix, '')
let prefix = ''
let bucket = ''
if (!path) return {
export const pathSlice = path => {
path = path.replace(minioBrowserPrefix, "")
let prefix = ""
let bucket = ""
if (!path)
return {
bucket,
prefix
}
let objectIndex = path.indexOf('/', 1)
}
let objectIndex = path.indexOf("/", 1)
if (objectIndex == -1) {
bucket = path.slice(1)
return {
@@ -79,7 +81,29 @@ export const pathSlice = (path) => {
}
export const pathJoin = (bucket, prefix) => {
if (!prefix)
prefix = ''
return minioBrowserPrefix + '/' + bucket + '/' + prefix
if (!prefix) prefix = ""
return minioBrowserPrefix + "/" + bucket + "/" + prefix
}
export const getRandomAccessKey = () => {
const alphaNumericTable = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
let arr = new Uint8Array(20)
window.crypto.getRandomValues(arr)
const random = Array.prototype.map.call(arr, v => {
const i = v % alphaNumericTable.length
return alphaNumericTable.charAt(i)
})
return random.join("")
}
export const getRandomSecretKey = () => {
let arr = new Uint8Array(40)
window.crypto.getRandomValues(arr)
const binStr = Array.prototype.map
.call(arr, v => {
return String.fromCharCode(v)
})
.join("")
const base64Str = btoa(binStr)
return base64Str.replace(/\//g, "+").substr(0, 40)
}
+3 -6
View File
@@ -72,6 +72,9 @@ class Web {
Logout() {
storage.removeItem('token')
}
GetToken() {
return storage.getItem('token')
}
ServerInfo() {
return this.makeCall('ServerInfo')
}
@@ -99,12 +102,6 @@ class Web {
RemoveObject(args) {
return this.makeCall('RemoveObject', args)
}
GetAuth() {
return this.makeCall('GetAuth')
}
GenerateAuth() {
return this.makeCall('GenerateAuth')
}
SetAuth(args) {
return this.makeCall('SetAuth', args)
.then(res => {
+3
View File
@@ -43,6 +43,9 @@ header.fesl-row {
color: @dark-gray;
font-size: 14px;
}
& > .fesli-sort--active {
.opacity(0.5);
}
&:hover:not(.fesl-item-actions) {
background: lighten(@text-muted-color, 22%);
+37
View File
@@ -113,4 +113,41 @@
margin: 0;
vertical-align: top;
}
}
.loading {
position: absolute;
margin: auto;
left: 0;
right: 0;
top: 0;
bottom: 0;
border-top: 1px solid @loading-track-bg;
border-right: 1px solid @loading-track-bg;
border-bottom: 1px solid @loading-track-bg;
border-left: 1px solid @loading-point-bg;
transform: translateZ(0);
animation: loading 1.1s infinite linear;
border-radius: 50%;
width: 35px;
height: 35px;
margin-top: 30px;
}
@-webkit-keyframes loading {
0% {
transform: rotate(0deg);
}
100% {
transform: rotate(360deg);
}
}
@keyframes loading {
0% {
transform: rotate(0deg);
}
100% {
transform: rotate(360deg);
}
}
+6 -2
View File
@@ -190,8 +190,8 @@
----------------------------*/
.toggle-password {
position: absolute;
bottom: 30px;
right: 35px;
bottom: 0 ;
right: 0;
width: 30px;
height: 30px;
border: 1px solid #eee;
@@ -206,6 +206,10 @@
background: #eee;
}
}
.has-toggle-password {
position: relative;
}
//--------------------------
+7 -1
View File
@@ -100,4 +100,10 @@
List
--------------------------*/
@list-row-selected-bg: #fbf2bf;
@list-row-even-bg: #fafafa;
@list-row-even-bg: #fafafa;
/*--------------------------
Loading
---------------------------*/
@loading-track-bg: #eeeeee;
@loading-point-bg: #00303f;
+14259
View File
File diff suppressed because it is too large Load Diff
+6 -5
View File
@@ -29,8 +29,8 @@
"homepage": "https://github.com/minio/minio",
"devDependencies": {
"async": "^1.5.2",
"babel-cli": "^6.14.0",
"babel-core": "^6.14.0",
"babel-cli": "^6.26.0",
"babel-core": "^6.26.3",
"babel-jest": "^22.1.0",
"babel-loader": "^7.1.2",
"babel-plugin-syntax-object-rest-spread": "^6.13.0",
@@ -38,10 +38,10 @@
"babel-polyfill": "^6.23.0",
"babel-preset-es2015": "^6.14.0",
"babel-preset-react": "^6.11.1",
"babel-register": "^6.14.0",
"babel-register": "^6.26.0",
"copy-webpack-plugin": "^4.6.0",
"css-loader": "^0.23.1",
"enzyme": "^3.3.0",
"enzyme": "^3.10.0",
"enzyme-adapter-react-16": "^1.1.1",
"esformatter": "^0.10.0",
"esformatter-jsx": "^7.4.1",
@@ -68,11 +68,12 @@
"humanize": "0.0.9",
"identity-obj-proxy": "^3.0.0",
"json-loader": "^0.5.4",
"jwt-decode": "^2.2.0",
"local-storage-fallback": "^4.0.2",
"material-design-iconic-font": "^2.2.0",
"mime-db": "^1.25.0",
"mime-types": "^2.1.13",
"moment": "^2.15.1",
"moment": "^2.24.0",
"react": "^16.2.0",
"react-addons-test-utils": "^0.14.8",
"react-bootstrap": "^0.32.0",
+31 -31
View File
File diff suppressed because one or more lines are too long
+213 -224
View File
@@ -251,22 +251,10 @@ ajv@^6.1.0:
json-schema-traverse "^0.4.1"
uri-js "^4.2.2"
align-text@^0.1.1, align-text@^0.1.3:
version "0.1.4"
resolved "https://registry.yarnpkg.com/align-text/-/align-text-0.1.4.tgz#0cd90a561093f35d0a99256c22b7069433fad117"
dependencies:
kind-of "^3.0.2"
longest "^1.0.1"
repeat-string "^1.5.2"
alphanum-sort@^1.0.1, alphanum-sort@^1.0.2:
version "1.0.2"
resolved "https://registry.yarnpkg.com/alphanum-sort/-/alphanum-sort-1.0.2.tgz#97a1119649b211ad33691d9f9f486a8ec9fbe0a3"
amdefine@>=0.0.4:
version "1.0.1"
resolved "https://registry.yarnpkg.com/amdefine/-/amdefine-1.0.1.tgz#4a5282ac164729e93619bcfd3ad151f817ce91f5"
ansi-colors@^3.0.0:
version "3.2.3"
resolved "https://registry.yarnpkg.com/ansi-colors/-/ansi-colors-3.2.3.tgz#57d35b8686e851e2cc04c403f1c00203976a1813"
@@ -378,6 +366,11 @@ array-equal@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/array-equal/-/array-equal-1.0.0.tgz#8c2a5ef2472fd9ea742b04c77a75093ba2757c93"
array-filter@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/array-filter/-/array-filter-1.0.0.tgz#baf79e62e6ef4c2a4c0b831232daffec251f9d83"
integrity sha1-uveeYubvTCpMC4MSMtr/7CUfnYM=
array-flatten@1.1.1:
version "1.1.1"
resolved "https://registry.yarnpkg.com/array-flatten/-/array-flatten-1.1.1.tgz#9a5f699051b1e7073328f2a008968b64ea2955d2"
@@ -410,6 +403,15 @@ array-unique@^0.3.2:
resolved "https://registry.yarnpkg.com/array-unique/-/array-unique-0.3.2.tgz#a894b75d4bc4f6cd679ef3244a9fd8f46ae2d428"
integrity sha1-qJS3XUvE9s1nnvMkSp/Y9Gri1Cg=
array.prototype.flat@^1.2.1:
version "1.2.1"
resolved "https://registry.yarnpkg.com/array.prototype.flat/-/array.prototype.flat-1.2.1.tgz#812db8f02cad24d3fab65dd67eabe3b8903494a4"
integrity sha512-rVqIs330nLJvfC7JqYvEWwqVr5QjYF1ib02i3YJtR/fICO6527Tjpc/e4Mvmxh3GIePPreRXMdaGyC99YphWEw==
dependencies:
define-properties "^1.1.2"
es-abstract "^1.10.0"
function-bind "^1.1.1"
arrify@^1.0.0, arrify@^1.0.1:
version "1.0.1"
resolved "https://registry.yarnpkg.com/arrify/-/arrify-1.0.1.tgz#898508da2226f380df904728456849c1501a4b0d"
@@ -467,7 +469,7 @@ async-limiter@~1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/async-limiter/-/async-limiter-1.0.0.tgz#78faed8c3d074ab81f22b4e985d79e8738f720f8"
async@^1.4.0, async@^1.5.2:
async@^1.5.2:
version "1.5.2"
resolved "https://registry.yarnpkg.com/async/-/async-1.5.2.tgz#ec6a61ae56480c0c3cb241c95618e20892f9672a"
@@ -520,24 +522,25 @@ aws4@^1.6.0:
version "1.6.0"
resolved "https://registry.yarnpkg.com/aws4/-/aws4-1.6.0.tgz#83ef5ca860b2b32e4a0deedee8c771b9db57471e"
babel-cli@^6.14.0:
version "6.24.1"
resolved "https://registry.yarnpkg.com/babel-cli/-/babel-cli-6.24.1.tgz#207cd705bba61489b2ea41b5312341cf6aca2283"
babel-cli@^6.26.0:
version "6.26.0"
resolved "https://registry.yarnpkg.com/babel-cli/-/babel-cli-6.26.0.tgz#502ab54874d7db88ad00b887a06383ce03d002f1"
integrity sha1-UCq1SHTX24itALiHoGODzgPQAvE=
dependencies:
babel-core "^6.24.1"
babel-polyfill "^6.23.0"
babel-register "^6.24.1"
babel-runtime "^6.22.0"
commander "^2.8.1"
convert-source-map "^1.1.0"
babel-core "^6.26.0"
babel-polyfill "^6.26.0"
babel-register "^6.26.0"
babel-runtime "^6.26.0"
commander "^2.11.0"
convert-source-map "^1.5.0"
fs-readdir-recursive "^1.0.0"
glob "^7.0.0"
lodash "^4.2.0"
output-file-sync "^1.1.0"
path-is-absolute "^1.0.0"
glob "^7.1.2"
lodash "^4.17.4"
output-file-sync "^1.1.2"
path-is-absolute "^1.0.1"
slash "^1.0.0"
source-map "^0.5.0"
v8flags "^2.0.10"
source-map "^0.5.6"
v8flags "^2.1.1"
optionalDependencies:
chokidar "^1.6.1"
@@ -581,29 +584,30 @@ babel-core@^6.0.0, babel-core@^6.26.0:
slash "^1.0.0"
source-map "^0.5.6"
babel-core@^6.14.0, babel-core@^6.24.1:
version "6.25.0"
resolved "https://registry.yarnpkg.com/babel-core/-/babel-core-6.25.0.tgz#7dd42b0463c742e9d5296deb3ec67a9322dad729"
babel-core@^6.26.3:
version "6.26.3"
resolved "https://registry.yarnpkg.com/babel-core/-/babel-core-6.26.3.tgz#b2e2f09e342d0f0c88e2f02e067794125e75c207"
integrity sha512-6jyFLuDmeidKmUEb3NM+/yawG0M2bDZ9Z1qbZP59cyHLz8kYGKYwpJP0UwUKKUiTRNvxfLesJnTedqczP7cTDA==
dependencies:
babel-code-frame "^6.22.0"
babel-generator "^6.25.0"
babel-code-frame "^6.26.0"
babel-generator "^6.26.0"
babel-helpers "^6.24.1"
babel-messages "^6.23.0"
babel-register "^6.24.1"
babel-runtime "^6.22.0"
babel-template "^6.25.0"
babel-traverse "^6.25.0"
babel-types "^6.25.0"
babylon "^6.17.2"
convert-source-map "^1.1.0"
debug "^2.1.1"
json5 "^0.5.0"
lodash "^4.2.0"
minimatch "^3.0.2"
path-is-absolute "^1.0.0"
private "^0.1.6"
babel-register "^6.26.0"
babel-runtime "^6.26.0"
babel-template "^6.26.0"
babel-traverse "^6.26.0"
babel-types "^6.26.0"
babylon "^6.18.0"
convert-source-map "^1.5.1"
debug "^2.6.9"
json5 "^0.5.1"
lodash "^4.17.4"
minimatch "^3.0.4"
path-is-absolute "^1.0.1"
private "^0.1.8"
slash "^1.0.0"
source-map "^0.5.0"
source-map "^0.5.7"
babel-generator@^6.18.0, babel-generator@^6.26.0:
version "6.26.0"
@@ -618,19 +622,6 @@ babel-generator@^6.18.0, babel-generator@^6.26.0:
source-map "^0.5.6"
trim-right "^1.0.1"
babel-generator@^6.25.0:
version "6.25.0"
resolved "https://registry.yarnpkg.com/babel-generator/-/babel-generator-6.25.0.tgz#33a1af70d5f2890aeb465a4a7793c1df6a9ea9fc"
dependencies:
babel-messages "^6.23.0"
babel-runtime "^6.22.0"
babel-types "^6.25.0"
detect-indent "^4.0.0"
jsesc "^1.3.0"
lodash "^4.2.0"
source-map "^0.5.0"
trim-right "^1.0.1"
babel-helper-builder-react-jsx@^6.24.1:
version "6.24.1"
resolved "https://registry.yarnpkg.com/babel-helper-builder-react-jsx/-/babel-helper-builder-react-jsx-6.24.1.tgz#0ad7917e33c8d751e646daca4e77cc19377d2cbc"
@@ -997,6 +988,15 @@ babel-polyfill@6.23.0, babel-polyfill@^6.23.0:
core-js "^2.4.0"
regenerator-runtime "^0.10.0"
babel-polyfill@^6.26.0:
version "6.26.0"
resolved "https://registry.yarnpkg.com/babel-polyfill/-/babel-polyfill-6.26.0.tgz#379937abc67d7895970adc621f284cd966cf2153"
integrity sha1-N5k3q8Z9eJWXCtxiHyhM2WbPIVM=
dependencies:
babel-runtime "^6.26.0"
core-js "^2.5.0"
regenerator-runtime "^0.10.5"
babel-preset-es2015@^6.14.0:
version "6.24.1"
resolved "https://registry.yarnpkg.com/babel-preset-es2015/-/babel-preset-es2015-6.24.1.tgz#d44050d6bc2c9feea702aaf38d727a0210538939"
@@ -1050,18 +1050,6 @@ babel-preset-react@^6.11.1:
babel-plugin-transform-react-jsx-source "^6.22.0"
babel-preset-flow "^6.23.0"
babel-register@^6.14.0, babel-register@^6.24.1:
version "6.24.1"
resolved "https://registry.yarnpkg.com/babel-register/-/babel-register-6.24.1.tgz#7e10e13a2f71065bdfad5a1787ba45bca6ded75f"
dependencies:
babel-core "^6.24.1"
babel-runtime "^6.22.0"
core-js "^2.4.0"
home-or-tmp "^2.0.0"
lodash "^4.2.0"
mkdirp "^0.5.1"
source-map-support "^0.4.2"
babel-register@^6.26.0:
version "6.26.0"
resolved "https://registry.yarnpkg.com/babel-register/-/babel-register-6.26.0.tgz#6ed021173e2fcb486d7acb45c6009a856f647071"
@@ -1098,7 +1086,7 @@ babel-template@^6.16.0, babel-template@^6.26.0:
babylon "^6.18.0"
lodash "^4.17.4"
babel-template@^6.24.1, babel-template@^6.25.0:
babel-template@^6.24.1:
version "6.25.0"
resolved "https://registry.yarnpkg.com/babel-template/-/babel-template-6.25.0.tgz#665241166b7c2aa4c619d71e192969552b10c071"
dependencies:
@@ -1509,10 +1497,6 @@ camel-case@3.0.x:
no-case "^2.2.0"
upper-case "^1.1.1"
camelcase@^1.0.2:
version "1.2.1"
resolved "https://registry.yarnpkg.com/camelcase/-/camelcase-1.2.1.tgz#9bb5304d2e0b56698b2c758b08a3eaa9daa58a39"
camelcase@^4.1.0:
version "4.1.0"
resolved "https://registry.yarnpkg.com/camelcase/-/camelcase-4.1.0.tgz#d545635be1e33c542649c69173e5de6acfae34dd"
@@ -1540,13 +1524,6 @@ caseless@~0.12.0:
resolved "https://registry.yarnpkg.com/caseless/-/caseless-0.12.0.tgz#1b681c21ff84033c826543090689420d187151dc"
integrity sha1-G2gcIf+EAzyCZUMJBolCDRhxUdw=
center-align@^0.1.1:
version "0.1.3"
resolved "https://registry.yarnpkg.com/center-align/-/center-align-0.1.3.tgz#aa0d32629b6ee972200411cbd4461c907bc2b7ad"
dependencies:
align-text "^0.1.3"
lazy-cache "^1.0.3"
chain-function@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/chain-function/-/chain-function-1.0.0.tgz#0d4ab37e7e18ead0bdc47b920764118ce58733dc"
@@ -1695,14 +1672,6 @@ cli-width@^2.0.0:
resolved "https://registry.yarnpkg.com/cli-width/-/cli-width-2.2.0.tgz#ff19ede8a9a5e579324147b0c11f0fbcbabed639"
integrity sha1-/xnt6Kml5XkyQUewwR8PvLq+1jk=
cliui@^2.1.0:
version "2.1.0"
resolved "https://registry.yarnpkg.com/cliui/-/cliui-2.1.0.tgz#4b475760ff80264c762c3a1719032e91c7fea0d1"
dependencies:
center-align "^0.1.1"
right-align "^0.1.1"
wordwrap "0.0.2"
cliui@^4.0.0:
version "4.0.0"
resolved "https://registry.yarnpkg.com/cliui/-/cliui-4.0.0.tgz#743d4650e05f36d1ed2575b59638d87322bfbbcc"
@@ -1802,7 +1771,12 @@ commander@2.17.x, commander@~2.17.1:
resolved "https://registry.yarnpkg.com/commander/-/commander-2.17.1.tgz#bd77ab7de6de94205ceacc72f1716d29f20a77bf"
integrity sha512-wPMUt6FnH2yzG95SA6mzjQOEKUU3aLaDEmzs1ti+1E9h+CsrZghRlqEM/EJ4KscsQVG8uNN4uVreUeT8+drlgg==
commander@^2.2.0, commander@^2.8.1, commander@^2.9.0:
commander@^2.11.0, commander@~2.20.0:
version "2.20.0"
resolved "https://registry.yarnpkg.com/commander/-/commander-2.20.0.tgz#d58bb2b5c1ee8f87b0d340027e9e94e222c5a422"
integrity sha512-7j2y+40w61zy6YC2iRNpUe/NwhNyoXrYpHMrSunaMG64nRnaf96zO/KMQR4OyN/UnE5KLyEBnKHd4aG3rskjpQ==
commander@^2.2.0, commander@^2.9.0:
version "2.10.0"
resolved "https://registry.yarnpkg.com/commander/-/commander-2.10.0.tgz#e1f5d3245de246d1a5ca04702fa1ad1bd7e405fe"
dependencies:
@@ -1893,14 +1867,17 @@ content-type@~1.0.4:
resolved "https://registry.yarnpkg.com/content-type/-/content-type-1.0.4.tgz#e138cc75e040c727b1966fe5e5f8c9aee256fe3b"
integrity sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==
convert-source-map@^1.1.0:
version "1.5.0"
resolved "https://registry.yarnpkg.com/convert-source-map/-/convert-source-map-1.5.0.tgz#9acd70851c6d5dfdd93d9282e5edf94a03ff46b5"
convert-source-map@^1.4.0, convert-source-map@^1.5.0:
version "1.5.1"
resolved "https://registry.yarnpkg.com/convert-source-map/-/convert-source-map-1.5.1.tgz#b8278097b9bc229365de5c62cf5fcaed8b5599e5"
convert-source-map@^1.5.1:
version "1.6.0"
resolved "https://registry.yarnpkg.com/convert-source-map/-/convert-source-map-1.6.0.tgz#51b537a8c43e0f04dec1993bffcdd504e758ac20"
integrity sha512-eFu7XigvxdZ1ETfbgPBohgyQ/Z++C0eEhTor0qRwBw9unw+L0/6V8wkSuGgzdThkiS5lSpdptOQPD8Ak40a+7A==
dependencies:
safe-buffer "~5.1.1"
cookie-signature@1.0.6:
version "1.0.6"
resolved "https://registry.yarnpkg.com/cookie-signature/-/cookie-signature-1.0.6.tgz#e303a882b342cc3ee8ca513a79999734dab3ae2c"
@@ -2170,7 +2147,7 @@ date-now@^0.1.4:
version "0.1.4"
resolved "https://registry.yarnpkg.com/date-now/-/date-now-0.1.4.tgz#eaf439fd4d4848ad74e5cc7dbef200672b9e345b"
debug@2.6.9, debug@^2.1.2, debug@^2.2.0, debug@^2.3.3, debug@^2.6.8:
debug@2.6.9, debug@^2.1.2, debug@^2.2.0, debug@^2.3.3, debug@^2.6.8, debug@^2.6.9:
version "2.6.9"
resolved "https://registry.yarnpkg.com/debug/-/debug-2.6.9.tgz#5d128515df134ff327e90a4c93f4e077a536341f"
dependencies:
@@ -2187,7 +2164,7 @@ debug@^0.7.4:
version "0.7.4"
resolved "https://registry.yarnpkg.com/debug/-/debug-0.7.4.tgz#06e1ea8082c2cb14e39806e22e2f6f757f92af39"
debug@^2.1.1, debug@^2.1.3:
debug@^2.1.3:
version "2.6.8"
resolved "https://registry.yarnpkg.com/debug/-/debug-2.6.8.tgz#e731531ca2ede27d188222427da17821d68ff4fc"
dependencies:
@@ -2207,7 +2184,7 @@ debug@^4.1.0:
dependencies:
ms "^2.1.1"
decamelize@^1.0.0, decamelize@^1.1.1, decamelize@^1.1.2, decamelize@^1.2.0:
decamelize@^1.1.1, decamelize@^1.1.2, decamelize@^1.2.0:
version "1.2.0"
resolved "https://registry.yarnpkg.com/decamelize/-/decamelize-1.2.0.tgz#f6534d15148269b20352e7bee26f501f9a191290"
integrity sha1-9lNNFRSCabIDUue+4m9QH5oZEpA=
@@ -2627,26 +2604,32 @@ enzyme-to-json@^3.3.0:
dependencies:
lodash "^4.17.4"
enzyme@^3.3.0:
version "3.3.0"
resolved "https://registry.yarnpkg.com/enzyme/-/enzyme-3.3.0.tgz#0971abd167f2d4bf3f5bd508229e1c4b6dc50479"
enzyme@^3.10.0:
version "3.10.0"
resolved "https://registry.yarnpkg.com/enzyme/-/enzyme-3.10.0.tgz#7218e347c4a7746e133f8e964aada4a3523452f6"
integrity sha512-p2yy9Y7t/PFbPoTvrWde7JIYB2ZyGC+NgTNbVEGvZ5/EyoYSr9aG/2rSbVvyNvMHEhw9/dmGUJHWtfQIEiX9pg==
dependencies:
array.prototype.flat "^1.2.1"
cheerio "^1.0.0-rc.2"
function.prototype.name "^1.0.3"
has "^1.0.1"
function.prototype.name "^1.1.0"
has "^1.0.3"
html-element-map "^1.0.0"
is-boolean-object "^1.0.0"
is-callable "^1.1.3"
is-callable "^1.1.4"
is-number-object "^1.0.3"
is-regex "^1.0.4"
is-string "^1.0.4"
is-subset "^0.1.1"
lodash "^4.17.4"
object-inspect "^1.5.0"
lodash.escape "^4.0.1"
lodash.isequal "^4.5.0"
object-inspect "^1.6.0"
object-is "^1.0.1"
object.assign "^4.1.0"
object.entries "^1.0.4"
object.values "^1.0.4"
raf "^3.4.0"
rst-selector-parser "^2.2.3"
string.prototype.trim "^1.1.2"
errno@^0.1.1, errno@^0.1.3, errno@~0.1.7:
version "0.1.7"
@@ -2662,6 +2645,18 @@ error-ex@^1.2.0:
dependencies:
is-arrayish "^0.2.1"
es-abstract@^1.10.0, es-abstract@^1.5.0:
version "1.13.0"
resolved "https://registry.yarnpkg.com/es-abstract/-/es-abstract-1.13.0.tgz#ac86145fdd5099d8dd49558ccba2eaf9b88e24e9"
integrity sha512-vDZfg/ykNxQVwup/8E1BZhVzFfBxs9NqMzGcvIJrqg5k2/5Za2bWo40dK2J1pgLngZ7c+Shh8lwYtLGyrwPutg==
dependencies:
es-to-primitive "^1.2.0"
function-bind "^1.1.1"
has "^1.0.3"
is-callable "^1.1.4"
is-regex "^1.0.4"
object-keys "^1.0.12"
es-abstract@^1.5.1:
version "1.10.0"
resolved "https://registry.yarnpkg.com/es-abstract/-/es-abstract-1.10.0.tgz#1ecb36c197842a00d8ee4c2dfd8646bb97d60864"
@@ -2681,7 +2676,7 @@ es-abstract@^1.6.1:
is-callable "^1.1.3"
is-regex "^1.0.3"
es-to-primitive@^1.1.1:
es-to-primitive@^1.1.1, es-to-primitive@^1.2.0:
version "1.2.0"
resolved "https://registry.yarnpkg.com/es-to-primitive/-/es-to-primitive-1.2.0.tgz#edf72478033456e8dda8ef09e00ad9650707f377"
integrity sha512-qZryBOJjV//LaxLTV6UC//WewneB3LcXOL9NP++ozKVXsIIIpm/2c13UDiD9Jp2eThsecw9m3jPqDwTyobcdbg==
@@ -3364,27 +3359,28 @@ fstream-ignore@^1.0.5:
minimatch "^3.0.0"
fstream@^1.0.0, fstream@^1.0.10, fstream@^1.0.2:
version "1.0.11"
resolved "https://registry.yarnpkg.com/fstream/-/fstream-1.0.11.tgz#5c1fb1f117477114f0632a0eb4b71b3cb0fd3171"
integrity sha1-XB+x8RdHcRTwYyoOtLcbPLD9MXE=
version "1.0.12"
resolved "https://registry.yarnpkg.com/fstream/-/fstream-1.0.12.tgz#4e8ba8ee2d48be4f7d0de505455548eae5932045"
integrity sha512-WvJ193OHa0GHPEL+AycEJgxvBEwyfRkN1vhjca23OaPVMCaLCXTd5qAu82AjTcgP1UJmytkOKb63Ypde7raDIg==
dependencies:
graceful-fs "^4.1.2"
inherits "~2.0.0"
mkdirp ">=0.5 0"
rimraf "2"
function-bind@^1.1.0:
version "1.1.0"
resolved "https://registry.yarnpkg.com/function-bind/-/function-bind-1.1.0.tgz#16176714c801798e4e8f2cf7f7529467bb4a5771"
function-bind@^1.1.1:
function-bind@^1.0.2, function-bind@^1.1.1:
version "1.1.1"
resolved "https://registry.yarnpkg.com/function-bind/-/function-bind-1.1.1.tgz#a56899d3ea3c9bab874bb9773b7c5ede92f4895d"
integrity sha512-yIovAzMX49sF8Yl58fSCWJ5svSLuaibPxXQJFLmBObTuCr0Mf1KiPopGM9NiFjiYBCbfaa2Fh6breQ6ANVTI0A==
function.prototype.name@^1.0.3:
function-bind@^1.1.0:
version "1.1.0"
resolved "https://registry.yarnpkg.com/function-bind/-/function-bind-1.1.0.tgz#16176714c801798e4e8f2cf7f7529467bb4a5771"
function.prototype.name@^1.1.0:
version "1.1.0"
resolved "https://registry.yarnpkg.com/function.prototype.name/-/function.prototype.name-1.1.0.tgz#8bd763cc0af860a859cc5d49384d74b932cd2327"
integrity sha512-Bs0VRrTz4ghD8pTmbJQD1mZ8A/mN0ur/jGz+A6FBxPDUPkm1tNfF6bhTYPA7i7aF4lZJVr+OXTNNrnnIl58Wfg==
dependencies:
define-properties "^1.1.2"
function-bind "^1.1.1"
@@ -3471,7 +3467,19 @@ glob-parent@^3.1.0:
is-glob "^3.1.0"
path-dirname "^1.0.0"
glob@^7.0.0, glob@^7.1.1, glob@^7.1.2:
glob@^7.0.3, glob@^7.0.5:
version "7.1.3"
resolved "https://registry.yarnpkg.com/glob/-/glob-7.1.3.tgz#3960832d3f1574108342dafd3a67b332c0969df1"
integrity sha512-vcfuiIxogLV4DlGBHIUOwI0IbrJ8HWPc4MU7HzviGeNho/UJDfi6B5p3sHeWIQ0KGIU0Jpxi5ZHxemQfLkkAwQ==
dependencies:
fs.realpath "^1.0.0"
inflight "^1.0.4"
inherits "2"
minimatch "^3.0.4"
once "^1.3.0"
path-is-absolute "^1.0.0"
glob@^7.1.1, glob@^7.1.2:
version "7.1.2"
resolved "https://registry.yarnpkg.com/glob/-/glob-7.1.2.tgz#c19c9df9a028702d678612384a6552404c636d15"
dependencies:
@@ -3482,10 +3490,10 @@ glob@^7.0.0, glob@^7.1.1, glob@^7.1.2:
once "^1.3.0"
path-is-absolute "^1.0.0"
glob@^7.0.3, glob@^7.0.5, glob@^7.1.3:
version "7.1.3"
resolved "https://registry.yarnpkg.com/glob/-/glob-7.1.3.tgz#3960832d3f1574108342dafd3a67b332c0969df1"
integrity sha512-vcfuiIxogLV4DlGBHIUOwI0IbrJ8HWPc4MU7HzviGeNho/UJDfi6B5p3sHeWIQ0KGIU0Jpxi5ZHxemQfLkkAwQ==
glob@^7.1.3:
version "7.1.4"
resolved "https://registry.yarnpkg.com/glob/-/glob-7.1.4.tgz#aa608a2f6c577ad357e1ae5a5c26d9a8d1969255"
integrity sha512-hkLPepehmnKk41pUGm3sYxoFs/umurYfYJCerbXEyFIWcAzvpipAgVkBqqT9RBKMGjnq6kMuyYwha6csxbiM1A==
dependencies:
fs.realpath "^1.0.0"
inflight "^1.0.4"
@@ -3546,11 +3554,7 @@ globby@^7.1.1:
pify "^3.0.0"
slash "^1.0.0"
graceful-fs@^4.1.11, graceful-fs@^4.1.4:
version "4.1.11"
resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.1.11.tgz#0e8bdfe4d1ddb8854d64e04ea7c00e2a026e5658"
graceful-fs@^4.1.15, graceful-fs@^4.1.2:
graceful-fs@^4.1.11, graceful-fs@^4.1.15, graceful-fs@^4.1.2, graceful-fs@^4.1.4:
version "4.1.15"
resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.1.15.tgz#ffb703e1066e8a0eeaa4c8b80ba9253eeefbfb00"
integrity sha512-6uHUhOPEBgQ24HM+r6b/QwWfZq+yiFcipKFrOFiBEnWdy5sdzYoi+pJeQaPI5qOLRFqWmAXUPQNsielzdLoecA==
@@ -3570,14 +3574,15 @@ handle-thing@^2.0.0:
integrity sha512-d4sze1JNC454Wdo2fkuyzCr6aHcbL6PGGuFAz0Li/NcOm1tCHGnWDRmJP85dh9IhQErTc2svWFEX5xHIOo//kQ==
handlebars@^4.0.3:
version "4.0.11"
resolved "https://registry.yarnpkg.com/handlebars/-/handlebars-4.0.11.tgz#630a35dfe0294bc281edae6ffc5d329fc7982dcc"
version "4.1.2"
resolved "https://registry.yarnpkg.com/handlebars/-/handlebars-4.1.2.tgz#b6b37c1ced0306b221e094fc7aca3ec23b131b67"
integrity sha512-nvfrjqvt9xQ8Z/w0ijewdD/vvWDTOweBUm96NTr66Wfvo1mJenBLwcYmPs3TIBP5ruzYGD7Hx/DaM9RmhroGPw==
dependencies:
async "^1.4.0"
neo-async "^2.6.0"
optimist "^0.6.1"
source-map "^0.4.4"
source-map "^0.6.1"
optionalDependencies:
uglify-js "^2.6"
uglify-js "^3.1.4"
har-schema@^1.0.5:
version "1.0.5"
@@ -3667,7 +3672,7 @@ has-values@^1.0.0:
is-number "^3.0.0"
kind-of "^4.0.0"
has@^1.0.1:
has@^1.0.1, has@^1.0.3:
version "1.0.3"
resolved "https://registry.yarnpkg.com/has/-/has-1.0.3.tgz#722d7cbfc1f6aa8241f16dd814e011e1f41e8796"
integrity sha512-f2dvO0VU6Oej7RkWJGrehjbzMAjFp5/VKPp5tTpWIV4JHHZK1/BxbFRtf/siA2SWTe09caDmVtYYzWEIbBS4zw==
@@ -3781,6 +3786,13 @@ html-comment-regex@^1.1.0:
version "1.1.1"
resolved "https://registry.yarnpkg.com/html-comment-regex/-/html-comment-regex-1.1.1.tgz#668b93776eaae55ebde8f3ad464b307a4963625e"
html-element-map@^1.0.0:
version "1.0.1"
resolved "https://registry.yarnpkg.com/html-element-map/-/html-element-map-1.0.1.tgz#3c4fcb4874ebddfe4283b51c8994e7713782b592"
integrity sha512-BZSfdEm6n706/lBfXKWa4frZRZcT5k1cOusw95ijZsHlI+GdgY0v95h6IzO3iIDf2ROwq570YTwqNPqHcNMozw==
dependencies:
array-filter "^1.0.0"
html-encoding-sniffer@^1.0.2:
version "1.0.2"
resolved "https://registry.yarnpkg.com/html-encoding-sniffer/-/html-encoding-sniffer-1.0.2.tgz#e70d84b94da53aa375e11fe3a351be6642ca46f8"
@@ -4907,6 +4919,11 @@ jsprim@^1.2.2:
json-schema "0.2.3"
verror "1.10.0"
jwt-decode@^2.2.0:
version "2.2.0"
resolved "https://registry.yarnpkg.com/jwt-decode/-/jwt-decode-2.2.0.tgz#7d86bd56679f58ce6a84704a657dd392bba81a79"
integrity sha1-fYa9VmefWM5qhHBKZX3TkruoGnk=
keycode@^2.1.2:
version "2.1.9"
resolved "https://registry.yarnpkg.com/keycode/-/keycode-2.1.9.tgz#964a23c54e4889405b4861a5c9f0480d45141dfa"
@@ -4940,10 +4957,6 @@ kind-of@^6.0.0, kind-of@^6.0.2:
resolved "https://registry.yarnpkg.com/kind-of/-/kind-of-6.0.2.tgz#01146b36a6218e64e58f3a8d66de5d7fc6f6d051"
integrity sha512-s5kLOcnH0XqDO+FvuaLX8DDjZ18CGFk7VygH40QoKPUQhW4e2rvM0rwUq0t8IQDOwYSeLK01U90OjzBTme2QqA==
lazy-cache@^1.0.3:
version "1.0.4"
resolved "https://registry.yarnpkg.com/lazy-cache/-/lazy-cache-1.0.4.tgz#a1d78fc3a50474cb80845d3b3b6e1da49a446e8e"
lcid@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/lcid/-/lcid-1.0.0.tgz#308accafa0bc483a3867b4b6f2b9506251d1b835"
@@ -5109,6 +5122,11 @@ lodash.deburr@^3.0.0:
dependencies:
lodash._root "^3.0.0"
lodash.escape@^4.0.1:
version "4.0.1"
resolved "https://registry.yarnpkg.com/lodash.escape/-/lodash.escape-4.0.1.tgz#c9044690c21e04294beaa517712fded1fa88de98"
integrity sha1-yQRGkMIeBClL6qUXcS/e0fqI3pg=
lodash.flattendeep@^4.4.0:
version "4.4.0"
resolved "https://registry.yarnpkg.com/lodash.flattendeep/-/lodash.flattendeep-4.4.0.tgz#fb030917f86a3134e5bc9bec0d69e0013ddfedb2"
@@ -5128,6 +5146,11 @@ lodash.isequal@^3.0:
lodash._baseisequal "^3.0.0"
lodash._bindcallback "^3.0.0"
lodash.isequal@^4.5.0:
version "4.5.0"
resolved "https://registry.yarnpkg.com/lodash.isequal/-/lodash.isequal-4.5.0.tgz#415c4478f2bcc30120c22ce10ed3226f7d3e18e0"
integrity sha1-QVxEePK8wwEgwizhDtMib30+GOA=
lodash.isplainobject@^4.0.6:
version "4.0.6"
resolved "https://registry.yarnpkg.com/lodash.isplainobject/-/lodash.isplainobject-4.0.6.tgz#7c526a52d89b45c45cc690b88163be0497f550cb"
@@ -5176,10 +5199,6 @@ loglevel@^1.4.1:
resolved "https://registry.yarnpkg.com/loglevel/-/loglevel-1.6.1.tgz#e0fc95133b6ef276cdc8887cdaf24aa6f156f8fa"
integrity sha1-4PyVEztu8nbNyIh82vJKpvFW+Po=
longest@^1.0.1:
version "1.0.1"
resolved "https://registry.yarnpkg.com/longest/-/longest-1.0.1.tgz#30a0b2da38f73770e8294a0d22e6625ed77d0097"
loose-envify@^1.0.0, loose-envify@^1.1.0, loose-envify@^1.2.0, loose-envify@^1.3.1:
version "1.3.1"
resolved "https://registry.yarnpkg.com/loose-envify/-/loose-envify-1.3.1.tgz#d1a8ad33fa9ce0e713d65fdd0ac8b748d478c848"
@@ -5220,8 +5239,9 @@ lru-cache@^5.1.1:
yallist "^3.0.2"
macaddress@^0.2.8:
version "0.2.8"
resolved "https://registry.yarnpkg.com/macaddress/-/macaddress-0.2.8.tgz#5904dc537c39ec6dbefeae902327135fa8511f12"
version "0.2.9"
resolved "https://registry.yarnpkg.com/macaddress/-/macaddress-0.2.9.tgz#3579b8b9acd5b96b4553abf0f394185a86813cb3"
integrity sha512-k4F1JUof6cQXxNFzx3thLby4oJzXTXQueAOOts944Vqizn+Rjc2QNFenT9FJSLU1CH3PmrHRSyZs2E+Cqw+P2w==
make-dir@^1.0.0:
version "1.3.0"
@@ -5441,7 +5461,7 @@ minimatch@^3.0.0, minimatch@^3.0.2, minimatch@^3.0.3, minimatch@^3.0.4:
dependencies:
brace-expansion "^1.1.7"
minimist@0.0.8, minimist@~0.0.1:
minimist@0.0.8:
version "0.0.8"
resolved "https://registry.yarnpkg.com/minimist/-/minimist-0.0.8.tgz#857fcabfc3397d2625b8228262e86aa7a011b05d"
integrity sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0=
@@ -5456,6 +5476,11 @@ minimist@^0.1.0:
resolved "https://registry.yarnpkg.com/minimist/-/minimist-0.1.0.tgz#99df657a52574c21c9057497df742790b2b4c0de"
integrity sha1-md9lelJXTCHJBXSX33QnkLK0wN4=
minimist@~0.0.1:
version "0.0.10"
resolved "https://registry.yarnpkg.com/minimist/-/minimist-0.0.10.tgz#de3f98543dbf96082be48ad1a0c7cda836301dcf"
integrity sha1-3j+YVD2/lggr5IrRoMfNqDYwHc8=
minipass@^2.2.1, minipass@^2.3.4:
version "2.3.5"
resolved "https://registry.yarnpkg.com/minipass/-/minipass-2.3.5.tgz#cacebe492022497f656b0f0f51e2682a9ed2d848"
@@ -5517,9 +5542,10 @@ mkdirp@0.5.x, "mkdirp@>=0.5 0", mkdirp@^0.5.0, mkdirp@^0.5.1, mkdirp@~0.5.0, mkd
dependencies:
minimist "0.0.8"
moment@^2.15.1:
version "2.18.1"
resolved "https://registry.yarnpkg.com/moment/-/moment-2.18.1.tgz#c36193dd3ce1c2eed2adb7c802dbbc77a81b1c0f"
moment@^2.24.0:
version "2.24.0"
resolved "https://registry.yarnpkg.com/moment/-/moment-2.24.0.tgz#0d055d53f5052aa653c9f6eb68bb5d12bf5c2b5b"
integrity sha512-bV7f+6l2QigeBBZSM/6yTNq4P2fNpSWj/0e7jQcy87A8e7o2nAfP/34/2ky5Vw4B9S446EtIhodAzkFCcR4dQg==
"mout@>=0.9 <2.0":
version "1.0.0"
@@ -5617,10 +5643,10 @@ negotiator@0.6.1:
resolved "https://registry.yarnpkg.com/negotiator/-/negotiator-0.6.1.tgz#2b327184e8992101177b28563fb5e7102acd0ca9"
integrity sha1-KzJxhOiZIQEXeyhWP7XnECrNDKk=
neo-async@^2.5.0:
version "2.6.0"
resolved "https://registry.yarnpkg.com/neo-async/-/neo-async-2.6.0.tgz#b9d15e4d71c6762908654b5183ed38b753340835"
integrity sha512-MFh0d/Wa7vkKO3Y3LlacqAEeHK0mckVqzDieUKTT+KGxi+zIpeVsFxymkIiRpbpDziHc290Xr9A1O4Om7otoRA==
neo-async@^2.5.0, neo-async@^2.6.0:
version "2.6.1"
resolved "https://registry.yarnpkg.com/neo-async/-/neo-async-2.6.1.tgz#ac27ada66167fa8849a6addd837f6b189ad2081c"
integrity sha512-iyam8fBuCUpWeKPGpaNMetEocMt364qkCsfL9JuhjXX6dRnguRVOfk2GZaDpPjcOKiiXCPINZC1GczQ7iTq3Zw==
nice-try@^1.0.4:
version "1.0.5"
@@ -5848,8 +5874,9 @@ number-is-nan@^1.0.0:
integrity sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0=
nwmatcher@^1.4.3:
version "1.4.3"
resolved "https://registry.yarnpkg.com/nwmatcher/-/nwmatcher-1.4.3.tgz#64348e3b3d80f035b40ac11563d278f8b72db89c"
version "1.4.4"
resolved "https://registry.yarnpkg.com/nwmatcher/-/nwmatcher-1.4.4.tgz#2285631f34a95f0d0395cd900c96ed39b58f346e"
integrity sha512-3iuY4N5dhgMpCUrOVnuAdGrgxVqV2cJpM+XNccjR2DKOB1RUP0aA+wGXEiNziG/UKboFyGBIoKOaNlJxx8bciQ==
oauth-sign@~0.8.1, oauth-sign@~0.8.2:
version "0.8.2"
@@ -5872,9 +5899,10 @@ object-inspect@^1.1.0:
version "1.2.2"
resolved "https://registry.yarnpkg.com/object-inspect/-/object-inspect-1.2.2.tgz#c82115e4fcc888aea14d64c22e4f17f6a70d5e5a"
object-inspect@^1.5.0:
version "1.5.0"
resolved "https://registry.yarnpkg.com/object-inspect/-/object-inspect-1.5.0.tgz#9d876c11e40f485c79215670281b767488f9bfe3"
object-inspect@^1.6.0:
version "1.6.0"
resolved "https://registry.yarnpkg.com/object-inspect/-/object-inspect-1.6.0.tgz#c70b6cbf72f274aab4c34c0c82f5167bf82cf15b"
integrity sha512-GJzfBZ6DgDAmnuaM3104jR4s1Myxr3Y3zfIyN4z3UdqN69oSRacNK8UhnobDdC+7J2AHCjGwxQubNJfE70SXXQ==
object-is@^1.0.1:
version "1.0.1"
@@ -5965,6 +5993,7 @@ on-headers@~1.0.1:
once@^1.3.0, once@^1.3.1, once@^1.3.3, once@^1.4.0:
version "1.4.0"
resolved "https://registry.yarnpkg.com/once/-/once-1.4.0.tgz#583b1aa775961d4b113ac17d9c50baef9dd76bd1"
integrity sha1-WDsap3WWHUsROsF9nFC6753Xa9E=
dependencies:
wrappy "1"
@@ -6005,6 +6034,7 @@ opn@^5.1.0:
optimist@^0.6.1:
version "0.6.1"
resolved "https://registry.yarnpkg.com/optimist/-/optimist-0.6.1.tgz#da3ea74686fa21a19a111c326e90eb15a0196686"
integrity sha1-2j6nRob6IaGaERwybpDrFaAZZoY=
dependencies:
minimist "~0.0.1"
wordwrap "~0.0.2"
@@ -6066,9 +6096,10 @@ osenv@^0.1.4:
os-homedir "^1.0.0"
os-tmpdir "^1.0.0"
output-file-sync@^1.1.0:
output-file-sync@^1.1.2:
version "1.1.2"
resolved "https://registry.yarnpkg.com/output-file-sync/-/output-file-sync-1.1.2.tgz#d0a33eefe61a205facb90092e826598d5245ce76"
integrity sha1-0KM+7+YaIF+suQCS6CZZjVJFznY=
dependencies:
graceful-fs "^4.1.4"
mkdirp "^0.5.1"
@@ -6642,7 +6673,7 @@ private@^0.1.6:
version "0.1.7"
resolved "https://registry.yarnpkg.com/private/-/private-0.1.7.tgz#68ce5e8a1ef0a23bb570cc28537b5332aba63ef1"
private@^0.1.7:
private@^0.1.7, private@^0.1.8:
version "0.1.8"
resolved "https://registry.yarnpkg.com/private/-/private-0.1.8.tgz#2381edb3689f7a53d653190060fcf822d2f368ff"
@@ -7153,7 +7184,7 @@ regenerate@^1.2.1:
version "1.3.2"
resolved "https://registry.yarnpkg.com/regenerate/-/regenerate-1.3.2.tgz#d1941c67bad437e1be76433add5b385f95b19260"
regenerator-runtime@^0.10.0:
regenerator-runtime@^0.10.0, regenerator-runtime@^0.10.5:
version "0.10.5"
resolved "https://registry.yarnpkg.com/regenerator-runtime/-/regenerator-runtime-0.10.5.tgz#336c3efc1220adcedda2c9fab67b5a7955a33658"
@@ -7386,25 +7417,13 @@ ret@~0.1.10:
version "0.1.15"
resolved "https://registry.yarnpkg.com/ret/-/ret-0.1.15.tgz#b8a4825d5bdb1fc3f6f53c2bc33f81388681c7bc"
right-align@^0.1.1:
version "0.1.3"
resolved "https://registry.yarnpkg.com/right-align/-/right-align-0.1.3.tgz#61339b722fe6a3515689210d24e14c96148613ef"
dependencies:
align-text "^0.1.1"
rimraf@2, rimraf@^2.2.8, rimraf@^2.5.1, rimraf@^2.6.1, rimraf@^2.6.2:
rimraf@2, rimraf@^2.2.8, rimraf@^2.5.1, rimraf@^2.5.4, rimraf@^2.6.1, rimraf@^2.6.2:
version "2.6.3"
resolved "https://registry.yarnpkg.com/rimraf/-/rimraf-2.6.3.tgz#b2d104fe0d8fb27cf9e0a1cda8262dd3833c6cab"
integrity sha512-mwqeW5XsA2qAejG46gYdENaxXjx9onRNCfn7L0duuP4hCuTIi/QO7PDK07KJfp1d+izWPrzEJDcSqBa0OZQriA==
dependencies:
glob "^7.1.3"
rimraf@^2.5.4:
version "2.6.2"
resolved "https://registry.yarnpkg.com/rimraf/-/rimraf-2.6.2.tgz#2ed8150d24a16ea8651e6d6ef0f47c4158ce7a36"
dependencies:
glob "^7.0.5"
ripemd160@^2.0.0, ripemd160@^2.0.1:
version "2.0.1"
resolved "https://registry.yarnpkg.com/ripemd160/-/ripemd160-2.0.1.tgz#0f4584295c53a3628af7e6d79aca21ce57d1c6e7"
@@ -7774,12 +7793,6 @@ source-map-support@^0.4.15:
dependencies:
source-map "^0.5.6"
source-map-support@^0.4.2:
version "0.4.15"
resolved "https://registry.yarnpkg.com/source-map-support/-/source-map-support-0.4.15.tgz#03202df65c06d2bd8c7ec2362a193056fef8d3b1"
dependencies:
source-map "^0.5.6"
source-map-support@^0.5.0:
version "0.5.3"
resolved "https://registry.yarnpkg.com/source-map-support/-/source-map-support-0.5.3.tgz#2b3d5fff298cfa4d1afd7d4352d569e9a0158e76"
@@ -7799,17 +7812,7 @@ source-map-url@^0.4.0:
resolved "https://registry.yarnpkg.com/source-map-url/-/source-map-url-0.4.0.tgz#3e935d7ddd73631b97659956d55128e87b5084a3"
integrity sha1-PpNdfd1zYxuXZZlW1VEo6HtQhKM=
source-map@^0.4.4:
version "0.4.4"
resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.4.4.tgz#eba4f5da9c0dc999de68032d8b4f76173652036b"
dependencies:
amdefine ">=0.0.4"
source-map@^0.5.0, source-map@^0.5.3:
version "0.5.6"
resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.5.6.tgz#75ce38f52bf0733c5a7f0c118d81334a2bb5f412"
source-map@^0.5.6, source-map@~0.5.1, source-map@~0.5.6:
source-map@^0.5.3, source-map@^0.5.6, source-map@^0.5.7, source-map@~0.5.6:
version "0.5.7"
resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.5.7.tgz#8a039d2d1021d22d1ea14c80d8ea468ba2ef3fcc"
integrity sha1-igOdLRAh0i0eoUyA2OpGi6LvP8w=
@@ -7817,6 +7820,7 @@ source-map@^0.5.6, source-map@~0.5.1, source-map@~0.5.6:
source-map@^0.6.0, source-map@^0.6.1, source-map@~0.6.0, source-map@~0.6.1:
version "0.6.1"
resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.6.1.tgz#74722af32e9614e9c287a8d0bbde48b5e2f1a263"
integrity sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==
spdx-correct@^3.0.0:
version "3.1.0"
@@ -8002,6 +8006,15 @@ string-width@^2.0.0:
is-fullwidth-code-point "^2.0.0"
strip-ansi "^3.0.0"
string.prototype.trim@^1.1.2:
version "1.1.2"
resolved "https://registry.yarnpkg.com/string.prototype.trim/-/string.prototype.trim-1.1.2.tgz#d04de2c89e137f4d7d206f086b5ed2fae6be8cea"
integrity sha1-0E3iyJ4Tf019IG8Ia17S+ua+jOo=
dependencies:
define-properties "^1.1.2"
es-abstract "^1.5.0"
function-bind "^1.0.2"
string_decoder@^1.0.0, string_decoder@~1.0.3:
version "1.0.3"
resolved "https://registry.yarnpkg.com/string_decoder/-/string_decoder-1.0.3.tgz#0fc67d7c141825de94282dd536bec6b9bce860ab"
@@ -8028,15 +8041,11 @@ string_decoder@~1.1.1:
dependencies:
safe-buffer "~5.1.0"
stringstream@~0.0.4:
stringstream@~0.0.4, stringstream@~0.0.5:
version "0.0.6"
resolved "https://registry.yarnpkg.com/stringstream/-/stringstream-0.0.6.tgz#7880225b0d4ad10e30927d167a1d6f2fd3b33a72"
integrity sha512-87GEBAkegbBcweToUrdzf3eLhWNg06FJTebl4BVJz/JgWy8CvEr9dRtX5qWphiynMSQlxxi+QqN0z5T32SLlhA==
stringstream@~0.0.5:
version "0.0.5"
resolved "https://registry.yarnpkg.com/stringstream/-/stringstream-0.0.5.tgz#4e484cd4de5a0bbbee18e46307710a8a81621878"
strip-ansi@^3.0.0, strip-ansi@^3.0.1:
version "3.0.1"
resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-3.0.1.tgz#6a385fb8853d952d5ff05d0e8aaf94278dc63dcf"
@@ -8400,18 +8409,13 @@ uglify-js@3.4.x:
commander "~2.17.1"
source-map "~0.6.1"
uglify-js@^2.6:
version "2.8.29"
resolved "https://registry.yarnpkg.com/uglify-js/-/uglify-js-2.8.29.tgz#29c5733148057bb4e1f75df35b7a9cb72e6a59dd"
uglify-js@^3.1.4:
version "3.6.0"
resolved "https://registry.yarnpkg.com/uglify-js/-/uglify-js-3.6.0.tgz#704681345c53a8b2079fb6cec294b05ead242ff5"
integrity sha512-W+jrUHJr3DXKhrsS7NUVxn3zqMOFn0hL/Ei6v0anCIMoKC93TjcflTagwIHLW7SfMFfiQuktQyFVCFHGUE0+yg==
dependencies:
source-map "~0.5.1"
yargs "~3.10.0"
optionalDependencies:
uglify-to-browserify "~1.0.0"
uglify-to-browserify@~1.0.0:
version "1.0.2"
resolved "https://registry.yarnpkg.com/uglify-to-browserify/-/uglify-to-browserify-1.0.2.tgz#6e0924d6bda6b5afe349e39a6d632850a0f882b7"
commander "~2.20.0"
source-map "~0.6.1"
uid-number@^0.0.6:
version "0.0.6"
@@ -8584,9 +8588,10 @@ v8-compile-cache@^2.0.2:
resolved "https://registry.yarnpkg.com/v8-compile-cache/-/v8-compile-cache-2.0.2.tgz#a428b28bb26790734c4fc8bc9fa106fccebf6a6c"
integrity sha512-1wFuMUIM16MDJRCrpbpuEPTUGmM5QMUg0cr3KFwra2XgOgFcPGDQHDh3CszSCD2Zewc/dh/pamNEW8CbfDebUw==
v8flags@^2.0.10:
v8flags@^2.1.1:
version "2.1.1"
resolved "https://registry.yarnpkg.com/v8flags/-/v8flags-2.1.1.tgz#aab1a1fa30d45f88dd321148875ac02c0b55e5b4"
integrity sha1-qrGh+jDUX4jdMhFIh1rALAtV5bQ=
dependencies:
user-home "^1.1.1"
@@ -8847,17 +8852,10 @@ wide-align@^1.1.0:
dependencies:
string-width "^1.0.2 || 2"
window-size@0.1.0:
version "0.1.0"
resolved "https://registry.yarnpkg.com/window-size/-/window-size-0.1.0.tgz#5438cd2ea93b202efa3a19fe8887aee7c94f9c9d"
wordwrap@0.0.2:
version "0.0.2"
resolved "https://registry.yarnpkg.com/wordwrap/-/wordwrap-0.0.2.tgz#b79669bb42ecb409f83d583cad52ca17eaa1643f"
wordwrap@~0.0.2:
version "0.0.3"
resolved "https://registry.yarnpkg.com/wordwrap/-/wordwrap-0.0.3.tgz#a3d5da6cd5c0bc0008d37234bbaf1bed63059107"
integrity sha1-o9XabNXAvAAI03I0u68b7WMFkQc=
wordwrap@~1.0.0:
version "1.0.0"
@@ -9011,12 +9009,3 @@ yargs@~1.2.6:
integrity sha1-nHtKgv1dWVsr8Xq23MQxNUMv40s=
dependencies:
minimist "^0.1.0"
yargs@~3.10.0:
version "3.10.0"
resolved "https://registry.yarnpkg.com/yargs/-/yargs-3.10.0.tgz#f7ee7bd857dd7c1d2d38c0e74efbd681d1431fd1"
dependencies:
camelcase "^1.0.2"
cliui "^2.1.0"
decamelize "^1.0.0"
window-size "0.1.0"
+118 -71
View File
@@ -34,8 +34,8 @@ import (
"github.com/tidwall/gjson"
"github.com/tidwall/sjson"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/cpu"
"github.com/minio/minio/pkg/disk"
"github.com/minio/minio/pkg/handlers"
@@ -44,6 +44,7 @@ import (
"github.com/minio/minio/pkg/mem"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/quick"
trace "github.com/minio/minio/pkg/trace"
)
const (
@@ -682,7 +683,7 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
// Start writing response to client
started = true
setCommonHeaders(w)
w.Header().Set("Content-Type", string(mimeJSON))
w.Header().Set(xhttp.ContentType, string(mimeJSON))
// Set 200 OK status
w.WriteHeader(200)
}
@@ -702,20 +703,20 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
var errorRespJSON []byte
if hr.errBody == "" {
errorRespJSON = encodeResponseJSON(getAPIErrorResponse(ctx, hr.apiErr,
r.URL.Path, w.Header().Get(responseRequestIDKey),
w.Header().Get(responseDeploymentIDKey)))
r.URL.Path, w.Header().Get(xhttp.AmzRequestID),
globalDeploymentID))
} else {
errorRespJSON = encodeResponseJSON(APIErrorResponse{
Code: hr.apiErr.Code,
Message: hr.errBody,
Resource: r.URL.Path,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
})
}
if !started {
setCommonHeaders(w)
w.Header().Set("Content-Type", string(mimeJSON))
w.Header().Set(xhttp.ContentType, string(mimeJSON))
w.WriteHeader(hr.apiErr.HTTPStatusCode)
}
w.Write(errorRespJSON)
@@ -786,6 +787,49 @@ func (a adminAPIHandlers) HealHandler(w http.ResponseWriter, r *http.Request) {
keepConnLive(w, respCh)
}
func (a adminAPIHandlers) BackgroundHealStatusHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "HealBackgroundStatus")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
return
}
// Check if this setup has an erasure coded backend.
if !globalIsXL {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrHealNotImplemented), r.URL)
return
}
var bgHealStates []madmin.BgHealState
// Get local heal status first
bgHealStates = append(bgHealStates, getLocalBackgroundHealStatus())
if globalIsDistXL {
// Get heal status from other peers
peersHealStates := globalNotificationSys.BackgroundHealStatus()
bgHealStates = append(bgHealStates, peersHealStates...)
}
// Aggregate healing result
var aggregatedHealStateResult = madmin.BgHealState{}
for _, state := range bgHealStates {
aggregatedHealStateResult.ScannedItemsCount += state.ScannedItemsCount
if aggregatedHealStateResult.LastHealActivity.Before(state.LastHealActivity) {
aggregatedHealStateResult.LastHealActivity = state.LastHealActivity
}
}
if err := json.NewEncoder(w).Encode(aggregatedHealStateResult); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
w.(http.Flusher).Flush()
}
// GetConfigHandler - GET /minio/admin/v1/config
// Get config.json of this minio setup.
func (a adminAPIHandlers) GetConfigHandler(w http.ResponseWriter, r *http.Request) {
@@ -941,8 +985,18 @@ func (a adminAPIHandlers) RemoveUser(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
accessKey := vars["accessKey"]
if err := globalIAMSys.DeleteUser(accessKey); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
// Notify all other MinIO peers to delete user.
for _, nerr := range globalNotificationSys.DeleteUser(accessKey) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
}
}
}
@@ -1007,8 +1061,8 @@ func (a adminAPIHandlers) SetUserStatus(w http.ResponseWriter, r *http.Request)
return
}
// Notify all other MinIO peers to reload users
for _, nerr := range globalNotificationSys.LoadUsers() {
// Notify all other MinIO peers to reload user.
for _, nerr := range globalNotificationSys.LoadUser(accessKey, false) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
@@ -1066,8 +1120,8 @@ func (a adminAPIHandlers) AddUser(w http.ResponseWriter, r *http.Request) {
return
}
// Notify all other MinIO peers to reload users
for _, nerr := range globalNotificationSys.LoadUsers() {
// Notify all other Minio peers to reload user
for _, nerr := range globalNotificationSys.LoadUser(accessKey, false) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
@@ -1084,7 +1138,7 @@ func (a adminAPIHandlers) ListCannedPolicies(w http.ResponseWriter, r *http.Requ
return
}
policies, err := globalIAMSys.ListCannedPolicies()
policies, err := globalIAMSys.ListPolicies()
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
@@ -1116,13 +1170,13 @@ func (a adminAPIHandlers) RemoveCannedPolicy(w http.ResponseWriter, r *http.Requ
return
}
if err := globalIAMSys.DeleteCannedPolicy(policyName); err != nil {
if err := globalIAMSys.DeletePolicy(policyName); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
// Notify all other MinIO peers to reload users
for _, nerr := range globalNotificationSys.LoadUsers() {
// Notify all other MinIO peers to delete policy
for _, nerr := range globalNotificationSys.DeletePolicy(policyName) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
@@ -1172,13 +1226,13 @@ func (a adminAPIHandlers) AddCannedPolicy(w http.ResponseWriter, r *http.Request
return
}
if err = globalIAMSys.SetCannedPolicy(policyName, *iamPolicy); err != nil {
if err = globalIAMSys.SetPolicy(policyName, *iamPolicy); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
// Notify all other MinIO peers to reload users
for _, nerr := range globalNotificationSys.LoadUsers() {
// Notify all other MinIO peers to reload policy
for _, nerr := range globalNotificationSys.LoadPolicy(policyName) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
@@ -1215,8 +1269,8 @@ func (a adminAPIHandlers) SetUserPolicy(w http.ResponseWriter, r *http.Request)
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
}
// Notify all other MinIO peers to reload users
for _, nerr := range globalNotificationSys.LoadUsers() {
// Notify all other Minio peers to reload user
for _, nerr := range globalNotificationSys.LoadUser(accessKey, false) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
@@ -1417,69 +1471,62 @@ func (a adminAPIHandlers) SetConfigKeysHandler(w http.ResponseWriter, r *http.Re
writeSuccessResponseHeadersOnly(w)
}
// UpdateAdminCredsHandler - POST /minio/admin/v1/config/credential
// TraceHandler - POST /minio/admin/v1/trace
// ----------
// Update admin credentials in a minio server
func (a adminAPIHandlers) UpdateAdminCredentialsHandler(w http.ResponseWriter,
r *http.Request) {
// The handler sends http trace to the connected HTTP client.
func (a adminAPIHandlers) TraceHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "HTTPTrace")
trcAll := r.URL.Query().Get("all") == "true"
ctx := newContext(r, w, "UpdateCredentialsHandler")
objectAPI := validateAdminReq(ctx, w, r)
if objectAPI == nil {
// Validate request signature.
adminAPIErr := checkAdminRequestAuthType(ctx, r, "")
if adminAPIErr != ErrNone {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(adminAPIErr), r.URL)
return
}
// Avoid setting new credentials when they are already passed
// by the environment. Deny if WORM is enabled.
if globalIsEnvCreds || globalWORMEnabled {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrMethodNotAllowed), r.URL)
return
}
// Avoid reusing tcp connection if read timeout is hit
// This is needed to make r.Context().Done() work as
// expected in case of read timeout
w.Header().Add(xhttp.Connection, "close")
if r.ContentLength > maxEConfigJSONSize || r.ContentLength == -1 {
// More than maxConfigSize bytes were available
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminConfigTooLarge), r.URL)
return
}
doneCh := make(chan struct{})
defer close(doneCh)
password := globalServerConfig.GetCredential().SecretKey
configBytes, err := madmin.DecryptData(password, io.LimitReader(r.Body, r.ContentLength))
// Trace Publisher and peer-trace-client uses nonblocking send and hence does not wait for slow receivers.
// Use buffered channel to take care of burst sends or slow w.Write()
traceCh := make(chan interface{}, 4000)
filter := func(entry interface{}) bool {
if trcAll {
return true
}
trcInfo := entry.(trace.Info)
return !strings.HasPrefix(trcInfo.ReqInfo.Path, minioReservedBucketPath)
}
remoteHosts := getRemoteHosts(globalEndpoints)
peers, err := getRestClients(remoteHosts)
if err != nil {
logger.LogIf(ctx, err)
writeCustomErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrAdminConfigBadJSON), err.Error(), r.URL)
return
}
globalHTTPTrace.Subscribe(traceCh, doneCh, filter)
// Decode request body
var req madmin.SetCredsReq
if err = json.Unmarshal(configBytes, &req); err != nil {
logger.LogIf(ctx, err)
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrRequestBodyParse), r.URL)
return
for _, peer := range peers {
peer.Trace(traceCh, doneCh, trcAll)
}
creds, err := auth.CreateCredentials(req.AccessKey, req.SecretKey)
if err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
enc := json.NewEncoder(w)
for {
select {
case entry := <-traceCh:
if err := enc.Encode(entry); err != nil {
return
}
w.(http.Flusher).Flush()
case <-r.Context().Done():
return
case <-GlobalServiceDoneCh:
return
}
}
// Acquire lock before updating global configuration.
globalServerConfigMu.Lock()
defer globalServerConfigMu.Unlock()
// Update local credentials in memory.
globalServerConfig.SetCredential(creds)
// Set active creds.
globalActiveCred = creds
if err = saveServerConfig(ctx, objectAPI, globalServerConfig); err != nil {
writeErrorResponseJSON(ctx, w, toAdminAPIErr(ctx, err), r.URL)
return
}
// Reply to the client before restarting minio server.
writeSuccessResponseHeadersOnly(w)
}
+8 -137
View File
@@ -20,7 +20,6 @@ import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"net/http"
@@ -100,6 +99,8 @@ var (
"enable": false,
"brokers": null,
"topic": "",
"queueDir": "",
"queueLimit": 0,
"tls": {
"enable": false,
"skipVerify": false,
@@ -193,7 +194,9 @@ var (
"webhook": {
"1": {
"enable": false,
"endpoint": ""
"endpoint": "",
"queueDir": "",
"queueLimit": 0
}
}
},
@@ -267,7 +270,9 @@ func prepareAdminXLTestBed() (*adminXLTestBed, error) {
initNSLock(isDistXL)
// Init global heal state
initAllHealState(globalIsXL)
if globalIsXL {
globalAllHealState = initHealState()
}
globalConfigSys = NewConfigSys()
@@ -298,60 +303,6 @@ func (atb *adminXLTestBed) TearDown() {
resetTestGlobals()
}
func (atb *adminXLTestBed) GenerateHealTestData(t *testing.T) {
// Create an object myobject under bucket mybucket.
bucketName := "mybucket"
err := atb.objLayer.MakeBucketWithLocation(context.Background(), bucketName, "")
if err != nil {
t.Fatalf("Failed to make bucket %s - %v", bucketName,
err)
}
// create some objects
{
objName := "myobject"
for i := 0; i < 10; i++ {
objectName := fmt.Sprintf("%s-%d", objName, i)
_, err = atb.objLayer.PutObject(context.Background(), bucketName, objectName,
mustGetPutObjReader(t, bytes.NewReader([]byte("hello")),
int64(len("hello")), "", ""), ObjectOptions{})
if err != nil {
t.Fatalf("Failed to create %s - %v", objectName,
err)
}
}
}
// create a multipart upload (incomplete)
{
objName := "mpObject"
uploadID, err := atb.objLayer.NewMultipartUpload(context.Background(), bucketName,
objName, ObjectOptions{})
if err != nil {
t.Fatalf("mp new error: %v", err)
}
_, err = atb.objLayer.PutObjectPart(context.Background(), bucketName, objName,
uploadID, 3, mustGetPutObjReader(t, bytes.NewReader(
[]byte("hello")), int64(len("hello")), "", ""), ObjectOptions{})
if err != nil {
t.Fatalf("mp put error: %v", err)
}
}
}
func (atb *adminXLTestBed) CleanupHealTestData(t *testing.T) {
bucketName := "mybucket"
objName := "myobject"
for i := 0; i < 10; i++ {
atb.objLayer.DeleteObject(context.Background(), bucketName,
fmt.Sprintf("%s-%d", objName, i))
}
atb.objLayer.DeleteBucket(context.Background(), bucketName)
}
// initTestObjLayer - Helper function to initialize an XL-based object
// layer and set globalObjectAPI.
func initTestXLObjLayer() (ObjectLayer, []string, error) {
@@ -599,86 +550,6 @@ func TestServiceRestartHandler(t *testing.T) {
testServicesCmdHandler(restartCmd, t)
}
// Test for service set creds management REST API.
func TestServiceSetCreds(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls. Note: In a
// single node setup, this degenerates to a simple function
// call under the hood.
globalMinioAddr = "127.0.0.1:9000"
credentials := globalServerConfig.GetCredential()
testCases := []struct {
AccessKey string
SecretKey string
EnvKeysSet bool
ExpectedStatusCode int
}{
// Bad secret key
{"minio", "minio", false, http.StatusBadRequest},
// Bad secret key set from the env
{"minio", "minio", true, http.StatusMethodNotAllowed},
// Good keys set from the env
{"minio", "minio123", true, http.StatusMethodNotAllowed},
// Successful operation should be the last one to
// not change server credentials during tests.
{"minio", "minio123", false, http.StatusOK},
}
for i, testCase := range testCases {
// Set or unset environement keys
globalIsEnvCreds = testCase.EnvKeysSet
// Construct setCreds request body
body, err := json.Marshal(madmin.SetCredsReq{
AccessKey: testCase.AccessKey,
SecretKey: testCase.SecretKey})
if err != nil {
t.Fatalf("JSONify err: %v", err)
}
ebody, err := madmin.EncryptData(credentials.SecretKey, body)
if err != nil {
t.Fatal(err)
}
// Construct setCreds request
req, err := getServiceCmdRequest(setCreds, credentials, ebody)
if err != nil {
t.Fatalf("Failed to build service status request %v", err)
}
rec := httptest.NewRecorder()
// Execute request
adminTestBed.router.ServeHTTP(rec, req)
// Check if the http code response is expected
if rec.Code != testCase.ExpectedStatusCode {
t.Errorf("Test %d: Wrong status code, expected = %d, found = %d", i+1, testCase.ExpectedStatusCode, rec.Code)
resp, _ := ioutil.ReadAll(rec.Body)
t.Errorf("Expected to receive %d status code but received %d. Body (%s)",
http.StatusOK, rec.Code, string(resp))
}
// If we got 200 OK, check if new credentials are really set
if rec.Code == http.StatusOK {
cred := globalServerConfig.GetCredential()
if cred.AccessKey != testCase.AccessKey {
t.Errorf("Test %d: Wrong access key, expected = %s, found = %s", i+1, testCase.AccessKey, cred.AccessKey)
}
if cred.SecretKey != testCase.SecretKey {
t.Errorf("Test %d: Wrong secret key, expected = %s, found = %s", i+1, testCase.SecretKey, cred.SecretKey)
}
}
}
}
// buildAdminRequest - helper function to build an admin API request.
func buildAdminRequest(queryVal url.Values, method, path string,
contentLength int64, bodySeeker io.ReadSeeker) (*http.Request, error) {
+146 -99
View File
@@ -96,22 +96,15 @@ type allHealState struct {
healSeqMap map[string]*healSequence
}
var (
// global server heal state
globalAllHealState allHealState
)
// initAllHealState - initialize healing apparatus
func initAllHealState(isErasureMode bool) {
if !isErasureMode {
return
}
globalAllHealState = allHealState{
// initHealState - initialize healing apparatus
func initHealState() *allHealState {
healState := &allHealState{
healSeqMap: make(map[string]*healSequence),
}
go globalAllHealState.periodicHealSeqsClean()
go healState.periodicHealSeqsClean()
return healState
}
func (ahs *allHealState) periodicHealSeqsClean() {
@@ -138,6 +131,19 @@ func (ahs *allHealState) periodicHealSeqsClean() {
}
}
// getHealSequenceByToken - Retrieve a heal sequence by token. The second
// argument returns if a heal sequence actually exists.
func (ahs *allHealState) getHealSequenceByToken(token string) (h *healSequence, exists bool) {
ahs.Lock()
defer ahs.Unlock()
for _, healSeq := range ahs.healSeqMap {
if healSeq.clientToken == token {
return healSeq, true
}
}
return nil, false
}
// getHealSequence - Retrieve a heal sequence by path. The second
// argument returns if a heal sequence actually exists.
func (ahs *allHealState) getHealSequence(path string) (h *healSequence, exists bool) {
@@ -305,6 +311,14 @@ type healSequence struct {
// path is just pathJoin(bucket, objPrefix)
path string
// List of entities (format, buckets, objects) to heal
sourceCh chan string
// Report healing progress, false if this is a background
// healing since currently there is no entity which will
// receive realtime healing status
reportProgress bool
// time at which heal sequence was started
startTime time.Time
@@ -334,6 +348,12 @@ type healSequence struct {
// the last result index sent to client
lastSentResultIndex int64
// Number of total items scanned
scannedItemsCount int64
// The time of the last scan/heal activity
lastHealActivity time.Time
// Holds the request-info for logging
ctx context.Context
}
@@ -348,14 +368,15 @@ func newHealSequence(bucket, objPrefix, clientAddr string,
ctx := logger.SetReqInfo(context.Background(), reqInfo)
return &healSequence{
bucket: bucket,
objPrefix: objPrefix,
path: pathJoin(bucket, objPrefix),
startTime: UTCNow(),
clientToken: mustGetUUID(),
clientAddress: clientAddr,
forceStarted: forceStart,
settings: hs,
bucket: bucket,
objPrefix: objPrefix,
path: pathJoin(bucket, objPrefix),
reportProgress: true,
startTime: UTCNow(),
clientToken: mustGetUUID(),
clientAddress: clientAddr,
forceStarted: forceStart,
settings: hs,
currentStatus: healSequenceStatus{
Summary: healNotStartedStatus,
HealSettings: hs,
@@ -484,7 +505,11 @@ func (h *healSequence) healSequenceStart() {
h.currentStatus.StartTime = UTCNow()
h.currentStatus.updateLock.Unlock()
go h.traverseAndHeal()
if h.sourceCh == nil {
go h.traverseAndHeal()
} else {
go h.healFromSourceCh()
}
select {
case err, ok := <-h.traverseAndHealDoneCh:
@@ -519,6 +544,95 @@ func (h *healSequence) healSequenceStart() {
}
}
func (h *healSequence) queueHealTask(path string, healType madmin.HealItemType) error {
var respCh = make(chan healResult)
defer close(respCh)
// Send heal request
globalBackgroundHealing.queueHealTask(healTask{path: path, responseCh: respCh, opts: h.settings})
// Wait for answer and push result to the client
res := <-respCh
if !h.reportProgress {
return nil
}
res.result.Type = healType
if res.err != nil {
// Object might have been deleted, by the time heal
// was attempted, we should ignore this object and return success.
if isErrObjectNotFound(res.err) {
return nil
}
// Only report object error
if healType != madmin.HealItemObject {
return res.err
}
res.result.Detail = res.err.Error()
}
return h.pushHealResultItem(res.result)
}
func (h *healSequence) healItemsFromSourceCh() error {
h.lastHealActivity = UTCNow()
// Start healing the config prefix.
if err := h.healMinioSysMeta(minioConfigPrefix)(); err != nil {
logger.LogIf(h.ctx, err)
}
// Start healing the bucket config prefix.
if err := h.healMinioSysMeta(bucketConfigPrefix)(); err != nil {
logger.LogIf(h.ctx, err)
}
for path := range h.sourceCh {
var itemType madmin.HealItemType
switch {
case path == "/":
itemType = madmin.HealItemMetadata
case !strings.Contains(path, "/"):
itemType = madmin.HealItemBucket
default:
itemType = madmin.HealItemObject
}
if err := h.queueHealTask(path, itemType); err != nil {
logger.LogIf(h.ctx, err)
}
h.scannedItemsCount++
h.lastHealActivity = UTCNow()
}
return nil
}
func (h *healSequence) healFromSourceCh() {
if err := h.healItemsFromSourceCh(); err != nil {
h.traverseAndHealDoneCh <- err
}
close(h.traverseAndHealDoneCh)
}
func (h *healSequence) healItems() error {
// Start with format healing
if err := h.healDiskFormat(); err != nil {
return err
}
// Start healing the config prefix.
if err := h.healMinioSysMeta(minioConfigPrefix)(); err != nil {
return err
}
// Start healing the bucket config prefix.
if err := h.healMinioSysMeta(bucketConfigPrefix)(); err != nil {
return err
}
// Heal buckets and objects
return h.healBuckets()
}
// traverseAndHeal - traverses on-disk data and performs healing
// according to settings. At each "safe" point it also checks if an
// external quit signal has been received and quits if so. Since the
@@ -527,31 +641,10 @@ func (h *healSequence) healSequenceStart() {
// has to wait until a safe point is reached, such as between scanning
// two objects.
func (h *healSequence) traverseAndHeal() {
var err error
checkErr := func(f func() error) {
switch {
case err != nil:
return
case h.isQuitting():
if err := h.healItems(); err != nil {
if h.isQuitting() {
err = errHealStopSignalled
return
}
err = f()
}
// Start with format healing
checkErr(h.healDiskFormat)
// Start healing the config prefix.
checkErr(h.healMinioSysMeta(minioConfigPrefix))
// Start healing the bucket config prefix.
checkErr(h.healMinioSysMeta(bucketConfigPrefix))
// Heal buckets and objects
checkErr(h.healBuckets)
if err != nil {
h.traverseAndHealDoneCh <- err
}
@@ -575,17 +668,14 @@ func (h *healSequence) healMinioSysMeta(metaPrefix string) func() error {
if h.isQuitting() {
return errHealStopSignalled
}
res, herr := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove, h.settings.ScanMode)
herr := h.queueHealTask(pathJoin(bucket, object), madmin.HealItemBucketMetadata)
// Object might have been deleted, by the time heal
// was attempted we ignore this object an move on.
if isErrObjectNotFound(herr) {
return nil
}
if herr != nil {
return herr
}
res.Type = madmin.HealItemBucketMetadata
return h.pushHealResultItem(res)
return herr
})
}
}
@@ -603,26 +693,7 @@ func (h *healSequence) healDiskFormat() error {
return errServerNotInitialized
}
res, err := objectAPI.HealFormat(h.ctx, h.settings.DryRun)
// return any error, ignore error returned when disks have
// already healed.
if err != nil && err != errNoHealRequired {
return errFnHealFromAPIErr(h.ctx, err)
}
// Healing succeeded notify the peers to reload format and re-initialize disks.
// We will not notify peers only if healing succeeded.
if err == nil {
for _, nerr := range globalNotificationSys.ReloadFormat(h.settings.DryRun) {
if nerr.Err != nil {
logger.GetReqInfo(h.ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(h.ctx, nerr.Err)
}
}
}
// Push format heal result
return h.pushHealResultItem(res)
return h.queueHealTask("/", madmin.HealItemMetadata)
}
// healBuckets - check for all buckets heal or just particular bucket.
@@ -664,13 +735,7 @@ func (h *healSequence) healBucket(bucket string) error {
return errServerNotInitialized
}
result, err := objectAPI.HealBucket(h.ctx, bucket, h.settings.DryRun, h.settings.Remove)
// handle heal-bucket error
if err != nil {
return err
}
if err = h.pushHealResultItem(result); err != nil {
if err := h.queueHealTask(bucket, madmin.HealItemBucket); err != nil {
return err
}
@@ -678,7 +743,7 @@ func (h *healSequence) healBucket(bucket string) error {
if h.objPrefix != "" {
// Check if an object named as the objPrefix exists,
// and if so heal it.
_, err = objectAPI.GetObjectInfo(h.ctx, bucket, h.objPrefix, ObjectOptions{})
_, err := objectAPI.GetObjectInfo(h.ctx, bucket, h.objPrefix, ObjectOptions{})
if err == nil {
if err = h.healObject(bucket, h.objPrefix); err != nil {
return err
@@ -689,8 +754,7 @@ func (h *healSequence) healBucket(bucket string) error {
return nil
}
if err = objectAPI.HealObjects(h.ctx, bucket,
h.objPrefix, h.healObject); err != nil {
if err := objectAPI.HealObjects(h.ctx, bucket, h.objPrefix, h.healObject); err != nil {
return errFnHealFromAPIErr(h.ctx, err)
}
return nil
@@ -702,28 +766,11 @@ func (h *healSequence) healObject(bucket, object string) error {
return errHealStopSignalled
}
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
}
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return errServerNotInitialized
}
hri, err := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove, h.settings.ScanMode)
if isErrObjectNotFound(err) {
return nil
}
if err != nil {
hri.Detail = err.Error()
}
return h.pushHealResultItem(hri)
return h.queueHealTask(pathJoin(bucket, object), madmin.HealItemObject)
}
+4 -2
View File
@@ -61,6 +61,8 @@ func registerAdminRouter(router *mux.Router, enableConfigOps, enableIAMOps bool)
adminV1Router.Methods(http.MethodPost).Path("/heal/{bucket}").HandlerFunc(httpTraceAll(adminAPI.HealHandler))
adminV1Router.Methods(http.MethodPost).Path("/heal/{bucket}/{prefix:.*}").HandlerFunc(httpTraceAll(adminAPI.HealHandler))
adminV1Router.Methods(http.MethodPost).Path("/background-heal/status").HandlerFunc(httpTraceAll(adminAPI.BackgroundHealStatusHandler))
/// Health operations
}
@@ -74,8 +76,6 @@ func registerAdminRouter(router *mux.Router, enableConfigOps, enableIAMOps bool)
/// Config operations
if enableConfigOps {
// Update credentials
adminV1Router.Methods(http.MethodPut).Path("/config/credential").HandlerFunc(httpTraceHdrs(adminAPI.UpdateAdminCredentialsHandler))
// Get config
adminV1Router.Methods(http.MethodGet).Path("/config").HandlerFunc(httpTraceHdrs(adminAPI.GetConfigHandler))
// Set config
@@ -118,6 +118,8 @@ func registerAdminRouter(router *mux.Router, enableConfigOps, enableIAMOps bool)
// Top locks
adminV1Router.Methods(http.MethodGet).Path("/top/locks").HandlerFunc(httpTraceHdrs(adminAPI.TopLocksHandler))
// HTTP Trace
adminV1Router.Methods(http.MethodGet).Path("/trace").HandlerFunc(adminAPI.TraceHandler)
// If none of the routes match, return error.
adminV1Router.NotFoundHandler = http.HandlerFunc(httpTraceHdrs(notFoundHandlerJSON))
}
-7
View File
@@ -20,13 +20,6 @@ import (
"encoding/xml"
)
const (
// Response request id.
responseRequestIDKey = "x-amz-request-id"
// Deployment id.
responseDeploymentIDKey = "x-minio-deployment-id"
)
// ObjectIdentifier carries key name for the object to delete.
type ObjectIdentifier struct {
ObjectName string `xml:"Key"`
+13 -14
View File
@@ -27,7 +27,7 @@ import (
"github.com/aliyun/aliyun-oss-go-sdk/oss"
"google.golang.org/api/googleapi"
minio "github.com/minio/minio-go"
minio "github.com/minio/minio-go/v6"
"github.com/minio/minio/cmd/crypto"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
@@ -51,6 +51,7 @@ type APIErrorResponse struct {
Key string `xml:"Key,omitempty" json:"Key,omitempty"`
BucketName string `xml:"BucketName,omitempty" json:"BucketName,omitempty"`
Resource string
Region string `xml:"Region,omitempty" json:"Region,omitempty"`
RequestID string `xml:"RequestId" json:"RequestId"`
HostID string `xml:"HostId" json:"HostId"`
}
@@ -65,6 +66,7 @@ const (
ErrBadDigest
ErrEntityTooSmall
ErrEntityTooLarge
ErrPolicyTooLarge
ErrIncompleteBody
ErrInternalError
ErrInvalidAccessKeyID
@@ -398,6 +400,11 @@ var errorCodes = errorCodeMap{
Description: "Your proposed upload exceeds the maximum allowed object size.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrPolicyTooLarge: {
Code: "PolicyTooLarge",
Description: "Policy exceeds the maximum allowed document size.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrIncompleteBody: {
Code: "IncompleteBody",
Description: "You did not provide the number of bytes specified by the Content-Length HTTP header.",
@@ -873,16 +880,6 @@ var errorCodes = errorCodeMap{
Description: "Object name already exists as a directory.",
HTTPStatusCode: http.StatusConflict,
},
ErrReadQuorum: {
Code: "XMinioReadQuorum",
Description: "Multiple disk failures, unable to reconstruct data.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrWriteQuorum: {
Code: "XMinioWriteQuorum",
Description: "Multiple disks failures, unable to write data.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrInvalidObjectName: {
Code: "XMinioInvalidObjectName",
Description: "Object name contains unsupported characters.",
@@ -1479,7 +1476,6 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
if err == nil {
return ErrNone
}
// Verify if the underlying error is signature mismatch.
switch err {
case errInvalidArgument:
@@ -1529,6 +1525,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrKMSAuthFailure
case errOperationTimedOut, context.Canceled, context.DeadlineExceeded:
apiErr = ErrOperationTimedOut
case errDiskNotFound:
apiErr = ErrSlowDown
}
// Compression errors
@@ -1586,9 +1584,9 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
case InvalidPart:
apiErr = ErrInvalidPart
case InsufficientWriteQuorum:
apiErr = ErrWriteQuorum
apiErr = ErrSlowDown
case InsufficientReadQuorum:
apiErr = ErrReadQuorum
apiErr = ErrSlowDown
case UnsupportedDelimiter:
apiErr = ErrNotImplemented
case InvalidMarkerPrefixCombination:
@@ -1738,6 +1736,7 @@ func getAPIErrorResponse(ctx context.Context, err APIError, resource, requestID,
BucketName: reqInfo.BucketName,
Key: reqInfo.ObjectName,
Resource: resource,
Region: globalServerConfig.GetRegion(),
RequestID: requestID,
HostID: hostID,
}
+2 -2
View File
@@ -39,8 +39,8 @@ var toAPIErrorTests = []struct {
{err: ObjectNameInvalid{}, errCode: ErrInvalidObjectName},
{err: InvalidUploadID{}, errCode: ErrNoSuchUpload},
{err: InvalidPart{}, errCode: ErrInvalidPart},
{err: InsufficientReadQuorum{}, errCode: ErrReadQuorum},
{err: InsufficientWriteQuorum{}, errCode: ErrWriteQuorum},
{err: InsufficientReadQuorum{}, errCode: ErrSlowDown},
{err: InsufficientWriteQuorum{}, errCode: ErrSlowDown},
{err: UnsupportedDelimiter{}, errCode: ErrNotImplemented},
{err: InvalidMarkerPrefixCombination{}, errCode: ErrNotImplemented},
{err: InvalidUploadIDKeyCombination{}, errCode: ErrNotImplemented},
+11 -10
View File
@@ -26,6 +26,7 @@ import (
"time"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
)
// Returns a hexadecimal representation of time at the
@@ -36,13 +37,13 @@ func mustGetRequestID(t time.Time) string {
// Write http common headers
func setCommonHeaders(w http.ResponseWriter) {
w.Header().Set("Server", "MinIO/"+ReleaseTag)
w.Header().Set(xhttp.ServerInfo, "MinIO/"+ReleaseTag)
// Set `x-amz-bucket-region` only if region is set on the server
// by default minio uses an empty region.
if region := globalServerConfig.GetRegion(); region != "" {
w.Header().Set("X-Amz-Bucket-Region", region)
w.Header().Set(xhttp.AmzBucketRegion, region)
}
w.Header().Set("Accept-Ranges", "bytes")
w.Header().Set(xhttp.AcceptRanges, "bytes")
// Remove sensitive information
crypto.RemoveSensitiveHeaders(w.Header())
@@ -72,23 +73,23 @@ func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, rs *HTTPRangeSp
// Set last modified time.
lastModified := objInfo.ModTime.UTC().Format(http.TimeFormat)
w.Header().Set("Last-Modified", lastModified)
w.Header().Set(xhttp.LastModified, lastModified)
// Set Etag if available.
if objInfo.ETag != "" {
w.Header()["ETag"] = []string{"\"" + objInfo.ETag + "\""}
w.Header()[xhttp.ETag] = []string{"\"" + objInfo.ETag + "\""}
}
if objInfo.ContentType != "" {
w.Header().Set("Content-Type", objInfo.ContentType)
w.Header().Set(xhttp.ContentType, objInfo.ContentType)
}
if objInfo.ContentEncoding != "" {
w.Header().Set("Content-Encoding", objInfo.ContentEncoding)
w.Header().Set(xhttp.ContentEncoding, objInfo.ContentEncoding)
}
if !objInfo.Expires.IsZero() {
w.Header().Set("Expires", objInfo.Expires.UTC().Format(http.TimeFormat))
w.Header().Set(xhttp.Expires, objInfo.Expires.UTC().Format(http.TimeFormat))
}
// Set all other user defined metadata.
@@ -124,10 +125,10 @@ func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, rs *HTTPRangeSp
}
// Set content length.
w.Header().Set("Content-Length", strconv.FormatInt(rangeLen, 10))
w.Header().Set(xhttp.ContentLength, strconv.FormatInt(rangeLen, 10))
if rs != nil {
contentRange := fmt.Sprintf("bytes %d-%d/%d", start, start+rangeLen-1, totalObjectSize)
w.Header().Set("Content-Range", contentRange)
w.Header().Set(xhttp.ContentRange, contentRange)
}
return nil
+15 -12
View File
@@ -22,9 +22,11 @@ import (
"net/http"
"net/url"
"path"
"strconv"
"strings"
"time"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/handlers"
)
@@ -521,8 +523,9 @@ func generateMultiDeleteResponse(quiet bool, deletedObjects []ObjectIdentifier,
func writeResponse(w http.ResponseWriter, statusCode int, response []byte, mType mimeType) {
setCommonHeaders(w)
if mType != mimeNone {
w.Header().Set("Content-Type", string(mType))
w.Header().Set(xhttp.ContentType, string(mType))
}
w.Header().Set(xhttp.ContentLength, strconv.Itoa(len(response)))
w.WriteHeader(statusCode)
if response != nil {
w.Write(response)
@@ -561,7 +564,7 @@ func writeSuccessNoContent(w http.ResponseWriter) {
// writeRedirectSeeOther writes Location header with http status 303
func writeRedirectSeeOther(w http.ResponseWriter, location string) {
w.Header().Set("Location", location)
w.Header().Set(xhttp.Location, location)
writeResponse(w, http.StatusSeeOther, nil, mimeNone)
}
@@ -575,12 +578,12 @@ func writeErrorResponse(ctx context.Context, w http.ResponseWriter, err APIError
case "SlowDown", "XMinioServerNotInitialized", "XMinioReadQuorum", "XMinioWriteQuorum":
// Set retry-after header to indicate user-agents to retry request after 120secs.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
w.Header().Set("Retry-After", "120")
w.Header().Set(xhttp.RetryAfter, "120")
case "AccessDenied":
// The request is from browser and also if browser
// is enabled we need to redirect.
if browser {
w.Header().Set("Location", minioReservedBucketPath+reqURL.Path)
w.Header().Set(xhttp.Location, minioReservedBucketPath+reqURL.Path)
w.WriteHeader(http.StatusTemporaryRedirect)
return
}
@@ -588,7 +591,7 @@ func writeErrorResponse(ctx context.Context, w http.ResponseWriter, err APIError
// Generate error response.
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path,
w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
w.Header().Get(xhttp.AmzRequestID), globalDeploymentID)
encodedErrorResponse := encodeResponse(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeXML)
}
@@ -601,7 +604,7 @@ func writeErrorResponseHeadersOnly(w http.ResponseWriter, err APIError) {
// useful for admin APIs.
func writeErrorResponseJSON(ctx context.Context, w http.ResponseWriter, err APIError, reqURL *url.URL) {
// Generate error response.
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path, w.Header().Get(responseRequestIDKey), w.Header().Get(responseDeploymentIDKey))
errorResponse := getAPIErrorResponse(ctx, err, reqURL.Path, w.Header().Get(xhttp.AmzRequestID), globalDeploymentID)
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeJSON)
}
@@ -619,8 +622,8 @@ func writeCustomErrorResponseJSON(ctx context.Context, w http.ResponseWriter, er
Resource: reqURL.Path,
BucketName: reqInfo.BucketName,
Key: reqInfo.ObjectName,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
}
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, err.HTTPStatusCode, encodedErrorResponse, mimeJSON)
@@ -635,12 +638,12 @@ func writeCustomErrorResponseXML(ctx context.Context, w http.ResponseWriter, err
case "SlowDown", "XMinioServerNotInitialized", "XMinioReadQuorum", "XMinioWriteQuorum":
// Set retry-after header to indicate user-agents to retry request after 120secs.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
w.Header().Set("Retry-After", "120")
w.Header().Set(xhttp.RetryAfter, "120")
case "AccessDenied":
// The request is from browser and also if browser
// is enabled we need to redirect.
if browser && globalIsBrowserEnabled {
w.Header().Set("Location", minioReservedBucketPath+reqURL.Path)
w.Header().Set(xhttp.Location, minioReservedBucketPath+reqURL.Path)
w.WriteHeader(http.StatusTemporaryRedirect)
return
}
@@ -653,8 +656,8 @@ func writeCustomErrorResponseXML(ctx context.Context, w http.ResponseWriter, err
Resource: reqURL.Path,
BucketName: reqInfo.BucketName,
Key: reqInfo.ObjectName,
RequestID: w.Header().Get(responseRequestIDKey),
HostID: w.Header().Get(responseDeploymentIDKey),
RequestID: w.Header().Get(xhttp.AmzRequestID),
HostID: globalDeploymentID,
}
encodedErrorResponse := encodeResponse(errorResponse)
+50 -43
View File
@@ -20,6 +20,7 @@ import (
"net/http"
"github.com/gorilla/mux"
xhttp "github.com/minio/minio/cmd/http"
)
// objectAPIHandler implements and provides http handlers for S3 API.
@@ -28,10 +29,12 @@ type objectAPIHandlers struct {
CacheAPI func() CacheObjectLayer
// Returns true of handlers should interpret encryption.
EncryptionEnabled func() bool
// Returns true if handlers allow SSE-KMS encryption headers.
AllowSSEKMS func() bool
}
// registerAPIRouter - registers S3 compatible APIs.
func registerAPIRouter(router *mux.Router, encryptionEnabled bool) {
func registerAPIRouter(router *mux.Router, encryptionEnabled, allowSSEKMS bool) {
// Initialize API.
api := objectAPIHandlers{
ObjectAPI: newObjectLayerFn,
@@ -39,6 +42,9 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled bool) {
EncryptionEnabled: func() bool {
return encryptionEnabled
},
AllowSSEKMS: func() bool {
return allowSSEKMS
},
}
// API Router
@@ -46,104 +52,105 @@ func registerAPIRouter(router *mux.Router, encryptionEnabled bool) {
var routers []*mux.Router
for _, domainName := range globalDomainNames {
routers = append(routers, apiRouter.Host("{bucket:.+}."+domainName).Subrouter())
routers = append(routers, apiRouter.Host("{bucket:.+}."+domainName+":{port:.*}").Subrouter())
}
routers = append(routers, apiRouter.PathPrefix("/{bucket}").Subrouter())
for _, bucket := range routers {
// Object operations
// HeadObject
bucket.Methods("HEAD").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.HeadObjectHandler))
bucket.Methods(http.MethodHead).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.HeadObjectHandler))
// CopyObjectPart
bucket.Methods("PUT").Path("/{object:.+}").HeadersRegexp("X-Amz-Copy-Source", ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodPut).Path("/{object:.+}").HeadersRegexp(xhttp.AmzCopySource, ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
// PutObjectPart
bucket.Methods("PUT").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectPartHandler)).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
// ListObjectPxarts
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.ListObjectPartsHandler)).Queries("uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.ListObjectPartsHandler)).Queries("uploadId", "{uploadId:.*}")
// CompleteMultipartUpload
bucket.Methods("POST").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.CompleteMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.CompleteMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
// NewMultipartUpload
bucket.Methods("POST").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.NewMultipartUploadHandler)).Queries("uploads", "")
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.NewMultipartUploadHandler)).Queries("uploads", "")
// AbortMultipartUpload
bucket.Methods("DELETE").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.AbortMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.AbortMultipartUploadHandler)).Queries("uploadId", "{uploadId:.*}")
// GetObjectACL - this is a dummy call.
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectACLHandler)).Queries("acl", "")
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectACLHandler)).Queries("acl", "")
// GetObjectTagging - this is a dummy call.
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectTaggingHandler)).Queries("tagging", "")
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectTaggingHandler)).Queries("tagging", "")
// SelectObjectContent
bucket.Methods("POST").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.SelectObjectContentHandler)).Queries("select", "").Queries("select-type", "2")
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.SelectObjectContentHandler)).Queries("select", "").Queries("select-type", "2")
// GetObject
bucket.Methods("GET").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectHandler))
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.GetObjectHandler))
// CopyObject
bucket.Methods("PUT").Path("/{object:.+}").HeadersRegexp("X-Amz-Copy-Source", ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectHandler))
bucket.Methods(http.MethodPut).Path("/{object:.+}").HeadersRegexp(xhttp.AmzCopySource, ".*?(\\/|%2F).*?").HandlerFunc(httpTraceAll(api.CopyObjectHandler))
// PutObject
bucket.Methods("PUT").Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectHandler))
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(httpTraceHdrs(api.PutObjectHandler))
// DeleteObject
bucket.Methods("DELETE").Path("/{object:.+}").HandlerFunc(httpTraceAll(api.DeleteObjectHandler))
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(httpTraceAll(api.DeleteObjectHandler))
/// Bucket operations
// GetBucketLocation
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketLocationHandler)).Queries("location", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketLocationHandler)).Queries("location", "")
// GetBucketPolicy
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketPolicyHandler)).Queries("policy", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketPolicyHandler)).Queries("policy", "")
// Dummy Bucket Calls
// GetBucketACL -- this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketACLHandler)).Queries("acl", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketACLHandler)).Queries("acl", "")
// GetBucketCors - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketCorsHandler)).Queries("cors", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketCorsHandler)).Queries("cors", "")
// GetBucketWebsiteHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketWebsiteHandler)).Queries("website", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketWebsiteHandler)).Queries("website", "")
// GetBucketVersioningHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketVersioningHandler)).Queries("versioning", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketVersioningHandler)).Queries("versioning", "")
// GetBucketAccelerateHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketAccelerateHandler)).Queries("accelerate", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketAccelerateHandler)).Queries("accelerate", "")
// GetBucketRequestPaymentHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketRequestPaymentHandler)).Queries("requestPayment", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketRequestPaymentHandler)).Queries("requestPayment", "")
// GetBucketLoggingHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketLoggingHandler)).Queries("logging", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketLoggingHandler)).Queries("logging", "")
// GetBucketLifecycleHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketLifecycleHandler)).Queries("lifecycle", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketLifecycleHandler)).Queries("lifecycle", "")
// GetBucketReplicationHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketReplicationHandler)).Queries("replication", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketReplicationHandler)).Queries("replication", "")
// GetBucketTaggingHandler - this is a dummy call.
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketTaggingHandler)).Queries("tagging", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketTaggingHandler)).Queries("tagging", "")
//DeleteBucketWebsiteHandler
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketWebsiteHandler)).Queries("website", "")
bucket.Methods(http.MethodDelete).HandlerFunc(httpTraceAll(api.DeleteBucketWebsiteHandler)).Queries("website", "")
// DeleteBucketTaggingHandler
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketTaggingHandler)).Queries("tagging", "")
bucket.Methods(http.MethodDelete).HandlerFunc(httpTraceAll(api.DeleteBucketTaggingHandler)).Queries("tagging", "")
// GetBucketNotification
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.GetBucketNotificationHandler)).Queries("notification", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.GetBucketNotificationHandler)).Queries("notification", "")
// ListenBucketNotification
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListenBucketNotificationHandler)).Queries("events", "{events:.*}")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.ListenBucketNotificationHandler)).Queries("events", "{events:.*}")
// ListMultipartUploads
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListMultipartUploadsHandler)).Queries("uploads", "")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.ListMultipartUploadsHandler)).Queries("uploads", "")
// ListObjectsV2
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListObjectsV2Handler)).Queries("list-type", "2")
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.ListObjectsV2Handler)).Queries("list-type", "2")
// ListObjectsV1 (Legacy)
bucket.Methods("GET").HandlerFunc(httpTraceAll(api.ListObjectsV1Handler))
bucket.Methods(http.MethodGet).HandlerFunc(httpTraceAll(api.ListObjectsV1Handler))
// PutBucketPolicy
bucket.Methods("PUT").HandlerFunc(httpTraceAll(api.PutBucketPolicyHandler)).Queries("policy", "")
bucket.Methods(http.MethodPut).HandlerFunc(httpTraceAll(api.PutBucketPolicyHandler)).Queries("policy", "")
// PutBucketNotification
bucket.Methods("PUT").HandlerFunc(httpTraceAll(api.PutBucketNotificationHandler)).Queries("notification", "")
bucket.Methods(http.MethodPut).HandlerFunc(httpTraceAll(api.PutBucketNotificationHandler)).Queries("notification", "")
// PutBucket
bucket.Methods("PUT").HandlerFunc(httpTraceAll(api.PutBucketHandler))
bucket.Methods(http.MethodPut).HandlerFunc(httpTraceAll(api.PutBucketHandler))
// HeadBucket
bucket.Methods("HEAD").HandlerFunc(httpTraceAll(api.HeadBucketHandler))
bucket.Methods(http.MethodHead).HandlerFunc(httpTraceAll(api.HeadBucketHandler))
// PostPolicy
bucket.Methods("POST").HeadersRegexp("Content-Type", "multipart/form-data*").HandlerFunc(httpTraceHdrs(api.PostPolicyBucketHandler))
bucket.Methods(http.MethodPost).HeadersRegexp(xhttp.ContentType, "multipart/form-data*").HandlerFunc(httpTraceHdrs(api.PostPolicyBucketHandler))
// DeleteMultipleObjects
bucket.Methods("POST").HandlerFunc(httpTraceAll(api.DeleteMultipleObjectsHandler)).Queries("delete", "")
bucket.Methods(http.MethodPost).HandlerFunc(httpTraceAll(api.DeleteMultipleObjectsHandler)).Queries("delete", "")
// DeleteBucketPolicy
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketPolicyHandler)).Queries("policy", "")
bucket.Methods(http.MethodDelete).HandlerFunc(httpTraceAll(api.DeleteBucketPolicyHandler)).Queries("policy", "")
// DeleteBucket
bucket.Methods("DELETE").HandlerFunc(httpTraceAll(api.DeleteBucketHandler))
bucket.Methods(http.MethodDelete).HandlerFunc(httpTraceAll(api.DeleteBucketHandler))
}
/// Root operation
// ListBuckets
apiRouter.Methods("GET").Path("/").HandlerFunc(httpTraceAll(api.ListBucketsHandler))
apiRouter.Methods(http.MethodGet).Path("/").HandlerFunc(httpTraceAll(api.ListBucketsHandler))
// If none of the routes match.
apiRouter.NotFoundHandler = http.HandlerFunc(httpTraceAll(notFoundHandler))
+54 -19
View File
@@ -29,6 +29,7 @@ import (
"strings"
jwtgo "github.com/dgrijalva/jwt-go"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/hash"
@@ -38,41 +39,41 @@ import (
// Verify if request has JWT.
func isRequestJWT(r *http.Request) bool {
return strings.HasPrefix(r.Header.Get("Authorization"), jwtAlgorithm)
return strings.HasPrefix(r.Header.Get(xhttp.Authorization), jwtAlgorithm)
}
// Verify if request has AWS Signature Version '4'.
func isRequestSignatureV4(r *http.Request) bool {
return strings.HasPrefix(r.Header.Get("Authorization"), signV4Algorithm)
return strings.HasPrefix(r.Header.Get(xhttp.Authorization), signV4Algorithm)
}
// Verify if request has AWS Signature Version '2'.
func isRequestSignatureV2(r *http.Request) bool {
return (!strings.HasPrefix(r.Header.Get("Authorization"), signV4Algorithm) &&
strings.HasPrefix(r.Header.Get("Authorization"), signV2Algorithm))
return (!strings.HasPrefix(r.Header.Get(xhttp.Authorization), signV4Algorithm) &&
strings.HasPrefix(r.Header.Get(xhttp.Authorization), signV2Algorithm))
}
// Verify if request has AWS PreSign Version '4'.
func isRequestPresignedSignatureV4(r *http.Request) bool {
_, ok := r.URL.Query()["X-Amz-Credential"]
_, ok := r.URL.Query()[xhttp.AmzCredential]
return ok
}
// Verify request has AWS PreSign Version '2'.
func isRequestPresignedSignatureV2(r *http.Request) bool {
_, ok := r.URL.Query()["AWSAccessKeyId"]
_, ok := r.URL.Query()[xhttp.AmzAccessKeyID]
return ok
}
// Verify if request has AWS Post policy Signature Version '4'.
func isRequestPostPolicySignatureV4(r *http.Request) bool {
return strings.Contains(r.Header.Get("Content-Type"), "multipart/form-data") &&
return strings.Contains(r.Header.Get(xhttp.ContentType), "multipart/form-data") &&
r.Method == http.MethodPost
}
// Verify if the request has AWS Streaming Signature Version '4'. This is only valid for 'PUT' operation.
func isRequestSignStreamingV4(r *http.Request) bool {
return r.Header.Get("x-amz-content-sha256") == streamingContentSHA256 &&
return r.Header.Get(xhttp.AmzContentSha256) == streamingContentSHA256 &&
r.Method == http.MethodPut
}
@@ -109,9 +110,9 @@ func getRequestAuthType(r *http.Request) authType {
return authTypeJWT
} else if isRequestPostPolicySignatureV4(r) {
return authTypePostPolicy
} else if _, ok := r.URL.Query()["Action"]; ok {
} else if _, ok := r.URL.Query()[xhttp.Action]; ok {
return authTypeSTS
} else if _, ok := r.Header["Authorization"]; !ok {
} else if _, ok := r.Header[xhttp.Authorization]; !ok {
return authTypeAnonymous
}
return authTypeUnknown
@@ -121,7 +122,7 @@ func getRequestAuthType(r *http.Request) authType {
// It does not accept presigned or JWT or anonymous requests.
func checkAdminRequestAuthType(ctx context.Context, r *http.Request, region string) APIErrorCode {
s3Err := ErrAccessDenied
if _, ok := r.Header["X-Amz-Content-Sha256"]; ok &&
if _, ok := r.Header[xhttp.AmzContentSha256]; ok &&
getRequestAuthType(r) == authTypeSigned && !skipContentSha256Cksum(r) {
// We only support admin credentials to access admin APIs.
@@ -148,11 +149,11 @@ func checkAdminRequestAuthType(ctx context.Context, r *http.Request, region stri
// Fetch the security token set by the client.
func getSessionToken(r *http.Request) (token string) {
token = r.Header.Get("X-Amz-Security-Token")
token = r.Header.Get(xhttp.AmzSecurityToken)
if token != "" {
return token
}
return r.URL.Query().Get("X-Amz-Security-Token")
return r.URL.Query().Get(xhttp.AmzSecurityToken)
}
// Fetch claims in the security token returned by the client, doesn't return
@@ -200,6 +201,39 @@ func getClaimsFromToken(r *http.Request) (map[string]interface{}, error) {
if _, ok = v.(string); !ok {
return nil, errInvalidAccessKeyID
}
if globalPolicyOPA == nil {
// If OPA is not set, session token should
// have a policy and its mandatory, reject
// requests without policy claim.
p, pok := claims[iampolicy.PolicyName]
if !pok {
return nil, errAuthentication
}
if _, pok = p.(string); !pok {
return nil, errAuthentication
}
sp, spok := claims[iampolicy.SessionPolicyName]
// Sub policy is optional, if not set return success.
if !spok {
return claims, nil
}
// Sub policy is set but its not a string, reject such requests
spStr, spok := sp.(string)
if !spok {
return nil, errAuthentication
}
// Looks like subpolicy is set and is a string, if set then its
// base64 encoded, decode it. Decoding fails reject such requests.
spBytes, err := base64.StdEncoding.DecodeString(spStr)
if err != nil {
// Base64 decoding fails, we should log to indicate
// something is malforming the request sent by client.
logger.LogIf(context.Background(), err)
return nil, errAuthentication
}
claims[iampolicy.SessionPolicyName] = string(spBytes)
}
return claims, nil
}
@@ -337,8 +371,8 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
contentMD5, contentSHA256 []byte
)
// Extract 'Content-Md5' if present.
if _, ok := r.Header["Content-Md5"]; ok {
contentMD5, err = base64.StdEncoding.Strict().DecodeString(r.Header.Get("Content-Md5"))
if _, ok := r.Header[xhttp.ContentMD5]; ok {
contentMD5, err = base64.StdEncoding.Strict().DecodeString(r.Header.Get(xhttp.ContentMD5))
if err != nil || len(contentMD5) == 0 {
return ErrInvalidDigest
}
@@ -347,14 +381,14 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
// Extract either 'X-Amz-Content-Sha256' header or 'X-Amz-Content-Sha256' query parameter (if V4 presigned)
// Do not verify 'X-Amz-Content-Sha256' if skipSHA256.
if skipSHA256 := skipContentSha256Cksum(r); !skipSHA256 && isRequestPresignedSignatureV4(r) {
if sha256Sum, ok := r.URL.Query()["X-Amz-Content-Sha256"]; ok && len(sha256Sum) > 0 {
if sha256Sum, ok := r.URL.Query()[xhttp.AmzContentSha256]; ok && len(sha256Sum) > 0 {
contentSHA256, err = hex.DecodeString(sha256Sum[0])
if err != nil {
return ErrContentSHA256Mismatch
}
}
} else if _, ok := r.Header["X-Amz-Content-Sha256"]; !skipSHA256 && ok {
contentSHA256, err = hex.DecodeString(r.Header.Get("X-Amz-Content-Sha256"))
} else if _, ok := r.Header[xhttp.AmzContentSha256]; !skipSHA256 && ok {
contentSHA256, err = hex.DecodeString(r.Header.Get(xhttp.AmzContentSha256))
if err != nil || len(contentSHA256) == 0 {
return ErrContentSHA256Mismatch
}
@@ -362,7 +396,8 @@ func isReqAuthenticated(ctx context.Context, r *http.Request, region string, sty
// Verify 'Content-Md5' and/or 'X-Amz-Content-Sha256' if present.
// The verification happens implicit during reading.
reader, err := hash.NewReader(r.Body, -1, hex.EncodeToString(contentMD5), hex.EncodeToString(contentSHA256), -1)
reader, err := hash.NewReader(r.Body, -1, hex.EncodeToString(contentMD5),
hex.EncodeToString(contentSHA256), -1, globalCLIContext.StrictS3Compat)
if err != nil {
return toAPIErrorCode(ctx, err)
}
+160
View File
@@ -0,0 +1,160 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/madmin"
)
// healTask represents what to heal along with options
// path: '/' => Heal disk formats along with metadata
// path: 'bucket/' or '/bucket/' => Heal bucket
// path: 'bucket/object' => Heal object
type healTask struct {
path string
opts madmin.HealOpts
// Healing response will be sent here
responseCh chan healResult
}
// healResult represents a healing result with a possible error
type healResult struct {
result madmin.HealResultItem
err error
}
// healRoutine receives heal tasks, to heal buckets, objects and format.json
type healRoutine struct {
tasks chan healTask
doneCh chan struct{}
}
// Add a new task in the tasks queue
func (h *healRoutine) queueHealTask(task healTask) {
h.tasks <- task
}
// Wait for heal requests and process them
func (h *healRoutine) run() {
ctx := context.Background()
for {
select {
case task, ok := <-h.tasks:
if !ok {
break
}
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
}
var res madmin.HealResultItem
var err error
bucket, object := urlPath2BucketObjectName(task.path)
switch {
case bucket == "" && object == "":
res, err = bgHealDiskFormat(ctx, task.opts)
case bucket != "" && object == "":
res, err = bgHealBucket(ctx, bucket, task.opts)
case bucket != "" && object != "":
res, err = bgHealObject(ctx, bucket, object, task.opts)
}
task.responseCh <- healResult{result: res, err: err}
case <-h.doneCh:
return
case <-GlobalServiceDoneCh:
return
}
}
}
func initHealRoutine() *healRoutine {
return &healRoutine{
tasks: make(chan healTask),
doneCh: make(chan struct{}),
}
}
func initBackgroundHealing() {
healBg := initHealRoutine()
go healBg.run()
globalBackgroundHealing = healBg
}
// bgHealDiskFormat - heals format.json, return value indicates if a
// failure error occurred.
func bgHealDiskFormat(ctx context.Context, opts madmin.HealOpts) (madmin.HealResultItem, error) {
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return madmin.HealResultItem{}, errServerNotInitialized
}
res, err := objectAPI.HealFormat(ctx, opts.DryRun)
// return any error, ignore error returned when disks have
// already healed.
if err != nil && err != errNoHealRequired {
return madmin.HealResultItem{}, err
}
// Healing succeeded notify the peers to reload format and re-initialize disks.
// We will not notify peers if healing is not required.
if err == nil {
for _, nerr := range globalNotificationSys.ReloadFormat(opts.DryRun) {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
}
}
}
return res, nil
}
// bghealBucket - traverses and heals given bucket
func bgHealBucket(ctx context.Context, bucket string, opts madmin.HealOpts) (madmin.HealResultItem, error) {
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return madmin.HealResultItem{}, errServerNotInitialized
}
return objectAPI.HealBucket(ctx, bucket, opts.DryRun, opts.Remove)
}
// bgHealObject - heal the given object and record result
func bgHealObject(ctx context.Context, bucket, object string, opts madmin.HealOpts) (madmin.HealResultItem, error) {
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return madmin.HealResultItem{}, errServerNotInitialized
}
return objectAPI.HealObject(ctx, bucket, object, opts.DryRun, opts.Remove, opts.ScanMode)
}
+11 -31
View File
@@ -32,33 +32,20 @@ type streamingBitrotWriter struct {
h hash.Hash
shardSize int64
canClose chan struct{} // Needed to avoid race explained in Close() call.
// Following two fields are used only to make sure that Write(p) is called such that
// len(p) is always the block size except the last block, i.e prevent programmer errors.
currentBlockIdx int
verifyTillIdx int
}
func (b *streamingBitrotWriter) Write(p []byte) (int, error) {
if b.currentBlockIdx < b.verifyTillIdx && int64(len(p)) != b.shardSize {
// All blocks except last should be of the length b.shardSize
logger.LogIf(context.Background(), errUnexpected)
return 0, errUnexpected
}
if len(p) == 0 {
return 0, nil
}
b.h.Reset()
b.h.Write(p)
hashBytes := b.h.Sum(nil)
n, err := b.iow.Write(hashBytes)
if n != len(hashBytes) {
logger.LogIf(context.Background(), err)
_, err := b.iow.Write(hashBytes)
if err != nil {
return 0, err
}
n, err = b.iow.Write(p)
b.currentBlockIdx++
return n, err
return b.iow.Write(p)
}
func (b *streamingBitrotWriter) Close() error {
@@ -78,16 +65,14 @@ func (b *streamingBitrotWriter) Close() error {
func newStreamingBitrotWriter(disk StorageAPI, volume, filePath string, length int64, algo BitrotAlgorithm, shardSize int64) io.WriteCloser {
r, w := io.Pipe()
h := algo.New()
bw := &streamingBitrotWriter{w, h, shardSize, make(chan struct{}), 0, int(length / shardSize)}
bw := &streamingBitrotWriter{w, h, shardSize, make(chan struct{})}
go func() {
bitrotSumsTotalSize := ceilFrac(length, shardSize) * int64(h.Size()) // Size used for storing bitrot checksums.
totalFileSize := bitrotSumsTotalSize + length
err := disk.CreateFile(volume, filePath, totalFileSize, r)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("storageDisk", disk.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
totalFileSize := int64(-1) // For compressed objects length will be unknown (represented by length=-1)
if length != -1 {
bitrotSumsTotalSize := ceilFrac(length, shardSize) * int64(h.Size()) // Size used for storing bitrot checksums.
totalFileSize = bitrotSumsTotalSize + length
}
err := disk.CreateFile(volume, filePath, totalFileSize, r)
r.CloseWithError(err)
close(bw.canClose)
}()
@@ -118,7 +103,7 @@ func (b *streamingBitrotReader) ReadAt(buf []byte, offset int64) (int, error) {
var err error
if offset%b.shardSize != 0 {
// Offset should always be aligned to b.shardSize
logger.LogIf(context.Background(), errUnexpected)
// Can never happen unless there are programmer bugs
return 0, errUnexpected
}
if b.rc == nil {
@@ -127,25 +112,20 @@ func (b *streamingBitrotReader) ReadAt(buf []byte, offset int64) (int, error) {
streamOffset := (offset/b.shardSize)*int64(b.h.Size()) + offset
b.rc, err = b.disk.ReadFileStream(b.volume, b.filePath, streamOffset, b.tillOffset-streamOffset)
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("storageDisk", b.disk.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return 0, err
}
}
if offset != b.currOffset {
logger.LogIf(context.Background(), errUnexpected)
// Can never happen unless there are programmer bugs
return 0, errUnexpected
}
b.h.Reset()
_, err = io.ReadFull(b.rc, b.hashBytes)
if err != nil {
logger.LogIf(context.Background(), err)
return 0, err
}
_, err = io.ReadFull(b.rc, buf)
if err != nil {
logger.LogIf(context.Background(), err)
return 0, err
}
b.h.Write(buf)
+2 -13
View File
@@ -31,19 +31,9 @@ type wholeBitrotWriter struct {
filePath string
shardSize int64 // This is the shard size of the erasure logic
hash.Hash // For bitrot hash
// Following two fields are used only to make sure that Write(p) is called such that
// len(p) is always the block size except the last block and prevent programmer errors.
currentBlockIdx int
lastBlockIdx int
}
func (b *wholeBitrotWriter) Write(p []byte) (int, error) {
if b.currentBlockIdx < b.lastBlockIdx && int64(len(p)) != b.shardSize {
// All blocks except last should be of the length b.shardSize
logger.LogIf(context.Background(), errUnexpected)
return 0, errUnexpected
}
err := b.disk.AppendFile(b.volume, b.filePath, p)
if err != nil {
logger.LogIf(context.Background(), err)
@@ -54,7 +44,6 @@ func (b *wholeBitrotWriter) Write(p []byte) (int, error) {
logger.LogIf(context.Background(), err)
return 0, err
}
b.currentBlockIdx++
return len(p), nil
}
@@ -63,8 +52,8 @@ func (b *wholeBitrotWriter) Close() error {
}
// Returns whole-file bitrot writer.
func newWholeBitrotWriter(disk StorageAPI, volume, filePath string, length int64, algo BitrotAlgorithm, shardSize int64) io.WriteCloser {
return &wholeBitrotWriter{disk, volume, filePath, shardSize, algo.New(), 0, int(length / shardSize)}
func newWholeBitrotWriter(disk StorageAPI, volume, filePath string, algo BitrotAlgorithm, shardSize int64) io.WriteCloser {
return &wholeBitrotWriter{disk, volume, filePath, shardSize, algo.New()}
}
// Implementation to verify bitrot for the whole file.
+1 -1
View File
@@ -120,7 +120,7 @@ func newBitrotWriter(disk StorageAPI, volume, filePath string, length int64, alg
if algo == HighwayHash256S {
return newStreamingBitrotWriter(disk, volume, filePath, length, algo, shardSize)
}
return newWholeBitrotWriter(disk, volume, filePath, length, algo, shardSize)
return newWholeBitrotWriter(disk, volume, filePath, algo, shardSize)
}
func newBitrotReader(disk StorageAPI, bucket string, filePath string, tillOffset int64, algo BitrotAlgorithm, sum []byte, shardSize int64) io.ReaderAt {
-7
View File
@@ -46,13 +46,6 @@ func validateListObjectsArgs(prefix, marker, delimiter, encodingType string, max
}
}
/// MinIO special conditions for ListObjects.
// Verify if delimiter is anything other than '/', which we do not support.
if delimiter != "" && delimiter != "/" {
return ErrNotImplemented
}
// Success.
return ErrNone
}
+50 -36
View File
@@ -30,8 +30,9 @@ import (
"github.com/gorilla/mux"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio-go/v6/pkg/set"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/dns"
"github.com/minio/minio/pkg/event"
@@ -263,16 +264,6 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
return
}
var s3Error APIErrorCode
if s3Error = checkRequestAuthType(ctx, r, policy.DeleteObjectAction, bucket, ""); s3Error != ErrNone {
// In the event access is denied, a 200 response should still be returned
// http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
if s3Error != ErrAccessDenied {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL, guessIsBrowserReq(r))
return
}
}
// Content-Length is required and should be non-zero
// http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
if r.ContentLength <= 0 {
@@ -319,42 +310,60 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
return
}
deleteObject := objectAPI.DeleteObject
deleteObjectsFn := objectAPI.DeleteObjects
if api.CacheAPI() != nil {
deleteObject = api.CacheAPI().DeleteObject
deleteObjectsFn = api.CacheAPI().DeleteObjects
}
var dErrs = make([]error, len(deleteObjects.Objects))
type delObj struct {
origIndex int
name string
}
var objectsToDelete []delObj
var dErrs = make([]APIErrorCode, len(deleteObjects.Objects))
for index, object := range deleteObjects.Objects {
// If the request is denied access, each item
// should be marked as 'AccessDenied'
if s3Error == ErrAccessDenied {
dErrs[index] = PrefixAccessDenied{
Bucket: bucket,
Object: object.ObjectName,
if dErrs[index] = checkRequestAuthType(ctx, r, policy.DeleteObjectAction, bucket, object.ObjectName); dErrs[index] != ErrNone {
if dErrs[index] == ErrSignatureDoesNotMatch || dErrs[index] == ErrInvalidAccessKeyID {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(dErrs[index]), r.URL, guessIsBrowserReq(r))
return
}
continue
}
dErrs[index] = deleteObject(ctx, bucket, object.ObjectName)
objectsToDelete = append(objectsToDelete, delObj{index, object.ObjectName})
}
toNames := func(input []delObj) (output []string) {
output = make([]string, len(input))
for i := range input {
output[i] = input[i].name
}
return
}
errs, err := deleteObjectsFn(ctx, bucket, toNames(objectsToDelete))
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
}
for i, obj := range objectsToDelete {
dErrs[obj.origIndex] = toAPIErrorCode(ctx, errs[i])
}
// Collect deleted objects and errors if any.
var deletedObjects []ObjectIdentifier
var deleteErrors []DeleteError
for index, err := range dErrs {
for index, errCode := range dErrs {
object := deleteObjects.Objects[index]
// Success deleted objects are collected separately.
if err == nil {
if errCode == ErrNone || errCode == ErrNoSuchKey {
deletedObjects = append(deletedObjects, object)
continue
}
if _, ok := err.(ObjectNotFound); ok {
// If the object is not found it should be
// accounted as deleted as per S3 spec.
deletedObjects = append(deletedObjects, object)
continue
}
apiErr := toAPIError(ctx, err)
apiErr := getAPIError(errCode)
// Error during delete should be collected separately.
deleteErrors = append(deleteErrors, DeleteError{
Code: apiErr.Code,
@@ -437,7 +446,8 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
}
// Make sure to add Location information here only for bucket
w.Header().Set("Location", getObjectLocation(r, globalDomainNames, bucket, ""))
w.Header().Set(xhttp.Location,
getObjectLocation(r, globalDomainNames, bucket, ""))
writeSuccessResponseHeadersOnly(w)
return
@@ -458,7 +468,7 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
}
// Make sure to add Location information here only for bucket
w.Header().Set("Location", path.Clean(r.URL.Path)) // Clean any trailing slashes.
w.Header().Set(xhttp.Location, path.Clean(r.URL.Path)) // Clean any trailing slashes.
writeSuccessResponseHeadersOnly(w)
}
@@ -490,6 +500,9 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
bucket := mux.Vars(r)["bucket"]
// To detect if the client has disconnected.
r.Body = &detectDisconnect{r.Body, r.Context().Done()}
// Require Content-Length to be set in the request
size := r.ContentLength
if size < 0 {
@@ -615,7 +628,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
hashReader, err := hash.NewReader(fileBody, fileSize, "", "", fileSize)
hashReader, err := hash.NewReader(fileBody, fileSize, "", "", fileSize, globalCLIContext.StrictS3Compat)
if err != nil {
logger.LogIf(ctx, err)
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
@@ -653,7 +666,8 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
info := ObjectInfo{Size: fileSize}
hashReader, err = hash.NewReader(reader, info.EncryptedSize(), "", "", fileSize) // do not try to verify encrypted content
// do not try to verify encrypted content
hashReader, err = hash.NewReader(reader, info.EncryptedSize(), "", "", fileSize, globalCLIContext.StrictS3Compat)
if err != nil {
writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL, guessIsBrowserReq(r))
return
@@ -669,8 +683,8 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
}
location := getObjectLocation(r, globalDomainNames, bucket, object)
w.Header()["ETag"] = []string{`"` + objInfo.ETag + `"`}
w.Header().Set("Location", location)
w.Header()[xhttp.ETag] = []string{`"` + objInfo.ETag + `"`}
w.Header().Set(xhttp.Location, location)
// Notify object created event.
defer sendEvent(eventArgs{
+1 -1
View File
@@ -762,7 +762,7 @@ func testAPIDeleteMultipleObjectsHandler(obj ObjectLayer, instanceType, bucketNa
apiRouter.ServeHTTP(rec, req)
// Assert the response code with the expected status.
if rec.Code != testCase.expectedRespStatus {
t.Errorf("Case %d: MinIO %s: Expected the response status to be `%d`, but instead found `%d`", i+1, instanceType, testCase.expectedRespStatus, rec.Code)
t.Errorf("Test %d: MinIO %s: Expected the response status to be `%d`, but instead found `%d`", i+1, instanceType, testCase.expectedRespStatus, rec.Code)
}
// read the response body.
+1 -1
View File
@@ -71,7 +71,7 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
// Error out if Content-Length is beyond allowed size.
if r.ContentLength > maxBucketPolicySize {
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrEntityTooLarge), r.URL, guessIsBrowserReq(r))
writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrPolicyTooLarge), r.URL, guessIsBrowserReq(r))
return
}
+4 -8
View File
@@ -29,7 +29,7 @@ import (
etcd "github.com/coreos/etcd/clientv3"
dns2 "github.com/miekg/dns"
"github.com/minio/cli"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio-go/v6/pkg/set"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/cmd/logger/target/console"
"github.com/minio/minio/cmd/logger/target/http"
@@ -161,6 +161,9 @@ func handleCommonCmdArgs(ctx *cli.Context) {
globalCertsCADir = &ConfigDir{path: filepath.Join(globalCertsDir.Get(), certsCADir)}
logger.FatalIf(mkdirAllIgnorePerm(globalCertsCADir.Get()), "Unable to create certs CA directory at %s", globalCertsCADir.Get())
// Check "compat" flag from command line argument.
globalCLIContext.StrictS3Compat = ctx.IsSet("compat") || ctx.GlobalIsSet("compat")
}
// Parses the given compression exclude list `extensions` or `content-types`.
@@ -208,13 +211,6 @@ func handleCommonEnvVars() {
globalIsBrowserEnabled = bool(browserFlag)
}
traceFile := os.Getenv("MINIO_HTTP_TRACE")
if traceFile != "" {
var err error
globalHTTPTraceFile, err = os.OpenFile(traceFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0660)
logger.FatalIf(err, "error opening file %s", traceFile)
}
etcdEndpointsEnv, ok := os.LookupEnv("MINIO_ETCD_ENDPOINTS")
if ok {
etcdEndpoints := strings.Split(etcdEndpointsEnv, ",")
+18 -38
View File
@@ -52,8 +52,19 @@ func readConfig(ctx context.Context, objAPI ObjectLayer, configFile string) ([]b
}
func deleteConfigEtcd(ctx context.Context, client *etcd.Client, configFile string) error {
_, err := client.Delete(ctx, configFile)
return err
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
_, err := client.Delete(timeoutCtx, configFile)
if err != nil {
if err == context.DeadlineExceeded {
return fmt.Errorf("etcd setup is unreachable, please check your endpoints %s",
client.Endpoints())
}
return fmt.Errorf("unexpected error %s returned by etcd setup, please check your endpoints %s",
err, client.Endpoints())
}
return nil
}
func deleteConfig(ctx context.Context, objAPI ObjectLayer, configFile string) error {
@@ -73,7 +84,7 @@ func saveConfigEtcd(ctx context.Context, client *etcd.Client, configFile string,
}
func saveConfig(ctx context.Context, objAPI ObjectLayer, configFile string, data []byte) error {
hashReader, err := hash.NewReader(bytes.NewReader(data), int64(len(data)), "", getSHA256Hash(data), int64(len(data)))
hashReader, err := hash.NewReader(bytes.NewReader(data), int64(len(data)), "", getSHA256Hash(data), int64(len(data)), globalCLIContext.StrictS3Compat)
if err != nil {
return err
}
@@ -88,9 +99,11 @@ func readConfigEtcd(ctx context.Context, client *etcd.Client, configFile string)
resp, err := client.Get(timeoutCtx, configFile)
if err != nil {
if err == context.DeadlineExceeded {
return nil, fmt.Errorf("etcd setup is unreachable, please check your endpoints %s", client.Endpoints())
return nil, fmt.Errorf("etcd setup is unreachable, please check your endpoints %s",
client.Endpoints())
}
return nil, fmt.Errorf("unexpected error %s returned by etcd setup, please check your endpoints %s", err, client.Endpoints())
return nil, fmt.Errorf("unexpected error %s returned by etcd setup, please check your endpoints %s",
err, client.Endpoints())
}
if resp.Count == 0 {
return nil, errConfigNotFound
@@ -103,40 +116,7 @@ func readConfigEtcd(ctx context.Context, client *etcd.Client, configFile string)
return nil, errConfigNotFound
}
// watchConfigEtcd - watches for changes on `configFile` on etcd and loads them.
func watchConfigEtcd(objAPI ObjectLayer, configFile string, loadCfgFn func(ObjectLayer) error) {
ctx, cancel := context.WithTimeout(context.Background(), defaultContextTimeout)
defer cancel()
for watchResp := range globalEtcdClient.Watch(ctx, configFile) {
for _, event := range watchResp.Events {
if event.IsModify() || event.IsCreate() {
loadCfgFn(objAPI)
}
}
}
}
func checkConfigEtcd(ctx context.Context, client *etcd.Client, configFile string) error {
timeoutCtx, cancel := context.WithTimeout(ctx, defaultContextTimeout)
defer cancel()
resp, err := client.Get(timeoutCtx, configFile)
if err != nil {
if err == context.DeadlineExceeded {
return fmt.Errorf("etcd setup is unreachable, please check your endpoints %s", client.Endpoints())
}
return fmt.Errorf("unexpected error %s returned by etcd setup, please check your endpoints %s", err, client.Endpoints())
}
if resp.Count == 0 {
return errConfigNotFound
}
return nil
}
func checkConfig(ctx context.Context, objAPI ObjectLayer, configFile string) error {
if globalEtcdClient != nil {
return checkConfigEtcd(ctx, globalEtcdClient, configFile)
}
if _, err := objAPI.GetObjectInfo(ctx, minioMetaBucket, configFile, ObjectOptions{}); err != nil {
// Treat object not found as config not found.
if isErrObjectNotFound(err) {
+3 -3
View File
@@ -325,7 +325,7 @@ func (s *serverConfig) TestNotificationTargets() error {
if !v.Enable {
continue
}
t, err := target.NewKafkaTarget(k, v)
t, err := target.NewKafkaTarget(k, v, GlobalServiceDoneCh)
if err != nil {
return fmt.Errorf("kafka(%s): %s", k, err.Error())
}
@@ -667,7 +667,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.Kafka {
if args.Enable {
newTarget, err := target.NewKafkaTarget(id, args)
newTarget, err := target.NewKafkaTarget(id, args, GlobalServiceDoneCh)
if err != nil {
logger.LogIf(context.Background(), err)
continue
@@ -767,7 +767,7 @@ func getNotificationTargets(config *serverConfig) *event.TargetList {
for id, args := range config.Notify.Webhook {
if args.Enable {
args.RootCAs = globalRootCAs
newTarget := target.NewWebhookTarget(id, args)
newTarget := target.NewWebhookTarget(id, args, GlobalServiceDoneCh)
if err := targetList.Add(newTarget); err != nil {
logger.LogIf(context.Background(), err)
continue
+2 -2
View File
@@ -200,10 +200,10 @@ func TestValidateConfig(t *testing.T) {
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "postgresql": { "1": { "enable": true, "connectionString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "" }}}}`, false},
// Test 16 - Test Kafka
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "kafka": { "1": { "enable": true, "brokers": null, "topic": "" } }}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "kafka": { "1": { "enable": true, "brokers": null, "topic": "", "queueDir": "", "queueLimit": 0 } }}}`, false},
// Test 17 - Test Webhook
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "webhook": { "1": { "enable": true, "endpoint": "" } }}}`, false},
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "webhook": { "1": { "enable": true, "endpoint": "", "queueDir": "", "queueLimit": 0} }}}`, false},
// Test 18 - Test MySQL
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "mysql": { "1": { "enable": true, "dsnString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "" }}}}`, false},
+24 -15
View File
@@ -29,7 +29,7 @@ import (
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/event"
"github.com/minio/minio/pkg/event/target"
"github.com/minio/minio/pkg/iam/policy"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/iam/validator"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/quick"
@@ -2413,6 +2413,7 @@ func migrateV27ToV28() error {
}
// Migrates ${HOME}/.minio/config.json to '<export_path>/.minio.sys/config/config.json'
// if etcd is configured then migrates /config/config.json to '<export_path>/.minio.sys/config/config.json'
func migrateConfigToMinioSys(objAPI ObjectLayer) (err error) {
// Construct path to config.json for the given bucket.
configFile := path.Join(minioConfigPrefix, minioConfigFile)
@@ -2423,10 +2424,14 @@ func migrateConfigToMinioSys(objAPI ObjectLayer) (err error) {
}
defer func() {
// Rename config.json to config.json.deprecated only upon
// success of this function.
if err == nil {
os.Rename(getConfigFile(), getConfigFile()+".deprecated")
if globalEtcdClient != nil {
deleteConfigEtcd(context.Background(), globalEtcdClient, configFile)
} else {
// Rename config.json to config.json.deprecated only upon
// success of this function.
os.Rename(getConfigFile(), getConfigFile()+".deprecated")
}
}
}()
@@ -2435,7 +2440,7 @@ func migrateConfigToMinioSys(objAPI ObjectLayer) (err error) {
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid data race between readConfig()
// and saveConfig().
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
objLock := globalNSMutex.NewNSLock(context.Background(), minioMetaBucket, transactionConfigFile)
if err = objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
@@ -2446,20 +2451,24 @@ func migrateConfigToMinioSys(objAPI ObjectLayer) (err error) {
return err
} // if errConfigNotFound proceed to migrate..
var configFiles = []string{
getConfigFile(),
getConfigFile() + ".deprecated",
configFile,
}
var config = &serverConfig{}
if _, err = Load(getConfigFile(), config); err != nil {
if !os.IsNotExist(err) {
return err
}
// Read from deprecate file as well if necessary.
if _, err = Load(getConfigFile()+".deprecated", config); err != nil {
for _, cfgFile := range configFiles {
if _, err = Load(cfgFile, config); err != nil {
if !os.IsNotExist(err) {
return err
}
// If all else fails simply initialize the server config.
return newSrvConfig(objAPI)
continue
}
break
}
if os.IsNotExist(err) {
// Initialize the server config, if no config exists.
return newSrvConfig(objAPI)
}
return saveServerConfig(context.Background(), objAPI, config)
}
@@ -2483,7 +2492,7 @@ func migrateMinioSysConfig(objAPI ObjectLayer) error {
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid data race between readConfig()
// and saveConfig().
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
objLock := globalNSMutex.NewNSLock(context.Background(), minioMetaBucket, transactionConfigFile)
if err := objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
+1 -1
View File
@@ -884,7 +884,7 @@ type serverConfigV32 struct {
} `json:"policy"`
}
// serverConfigV33 is just like version '32', removes clientID from NATS and MQTT, and adds queueDir, queueLimit with MQTT.
// serverConfigV33 is just like version '32', removes clientID from NATS and MQTT, and adds queueDir, queueLimit in notification targets.
type serverConfigV33 struct {
quick.Config `json:"-"` // ignore interfaces
+17 -48
View File
@@ -49,10 +49,6 @@ func saveServerConfig(ctx context.Context, objAPI ObjectLayer, config *serverCon
}
configFile := path.Join(minioConfigPrefix, minioConfigFile)
if globalEtcdClient != nil {
return saveConfigEtcd(ctx, globalEtcdClient, configFile, data)
}
// Create a backup of the current config
oldData, err := readConfig(ctx, objAPI, configFile)
if err == nil {
@@ -71,15 +67,8 @@ func saveServerConfig(ctx context.Context, objAPI ObjectLayer, config *serverCon
}
func readServerConfig(ctx context.Context, objAPI ObjectLayer) (*serverConfig, error) {
var configData []byte
var err error
configFile := path.Join(minioConfigPrefix, minioConfigFile)
if globalEtcdClient != nil {
configData, err = readConfigEtcd(ctx, globalEtcdClient, configFile)
} else {
configData, err = readConfig(ctx, objAPI, configFile)
}
configData, err := readConfig(ctx, objAPI, configFile)
if err != nil {
return nil, err
}
@@ -152,45 +141,25 @@ func initConfig(objAPI ObjectLayer) error {
return errServerNotInitialized
}
configFile := path.Join(minioConfigPrefix, minioConfigFile)
if globalEtcdClient != nil {
if err := checkConfigEtcd(context.Background(), globalEtcdClient, getConfigFile()); err != nil {
if err == errConfigNotFound {
// Migrates all configs at old location.
if err = migrateConfig(); err != nil {
return err
}
// Migrates etcd ${HOME}/.minio/config.json to '/config/config.json'
if err = migrateConfigToMinioSys(objAPI); err != nil {
return err
}
} else {
return err
}
}
// Watch config for changes and reloads them.
go watchConfigEtcd(objAPI, configFile, loadConfig)
} else {
if isFile(getConfigFile()) {
if err := migrateConfig(); err != nil {
return err
}
}
// Migrates ${HOME}/.minio/config.json or config.json.deprecated
// to '<export_path>/.minio.sys/config/config.json'
// ignore if the file doesn't exist.
if err := migrateConfigToMinioSys(objAPI); err != nil {
return err
}
// Migrates backend '<export_path>/.minio.sys/config/config.json' to latest version.
if err := migrateMinioSysConfig(objAPI); err != nil {
if isFile(getConfigFile()) {
if err := migrateConfig(); err != nil {
return err
}
}
// Migrates ${HOME}/.minio/config.json or config.json.deprecated
// to '<export_path>/.minio.sys/config/config.json'
// ignore if the file doesn't exist.
// If etcd is set then migrates /config/config.json
// to '<export_path>/.minio.sys/config/config.json'
if err := migrateConfigToMinioSys(objAPI); err != nil {
return err
}
// Migrates backend '<export_path>/.minio.sys/config/config.json' to latest version.
if err := migrateMinioSysConfig(objAPI); err != nil {
return err
}
return loadConfig(objAPI)
}
+20
View File
@@ -18,6 +18,7 @@ import (
"bytes"
"crypto/md5"
"encoding/base64"
"encoding/json"
"net/http"
"strings"
)
@@ -125,6 +126,25 @@ func (s3KMS) IsRequested(h http.Header) bool {
return false
}
// ParseHTTP parses the SSE-KMS headers and returns the SSE-KMS key ID
// and context, if present, on success.
func (s3KMS) ParseHTTP(h http.Header) (string, interface{}, error) {
algorithm := h.Get(SSEHeader)
if algorithm != SSEAlgorithmKMS {
return "", nil, ErrInvalidEncryptionMethod
}
contextStr, ok := h[SSEKmsContext]
if ok {
var context map[string]interface{}
if err := json.Unmarshal([]byte(contextStr[0]), &context); err != nil {
return "", nil, err
}
return h.Get(SSEKmsID), context, nil
}
return h.Get(SSEKmsID), nil, nil
}
var (
// SSEC represents AWS SSE-C. It provides functionality to handle
// SSE-C requests.
+50
View File
@@ -54,6 +54,56 @@ func TestKMSIsRequested(t *testing.T) {
}
}
var kmsParseHTTPTests = []struct {
Header http.Header
ShouldFail bool
}{
{Header: http.Header{}, ShouldFail: true}, // 0
{Header: http.Header{"X-Amz-Server-Side-Encryption": []string{"aws:kms"}}, ShouldFail: false}, // 1
{Header: http.Header{
"X-Amz-Server-Side-Encryption": []string{"aws:kms"},
"X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id": []string{"s3-007-293847485-724784"},
}, ShouldFail: false}, // 2
{Header: http.Header{
"X-Amz-Server-Side-Encryption": []string{"aws:kms"},
"X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id": []string{"s3-007-293847485-724784"},
"X-Amz-Server-Side-Encryption-Context": []string{"{}"},
}, ShouldFail: false}, // 3
{Header: http.Header{
"X-Amz-Server-Side-Encryption": []string{"aws:kms"},
"X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id": []string{"s3-007-293847485-724784"},
"X-Amz-Server-Side-Encryption-Context": []string{"{\"bucket\": \"some-bucket\"}"},
}, ShouldFail: false}, // 4
{Header: http.Header{
"X-Amz-Server-Side-Encryption": []string{"aws:kms"},
"X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id": []string{"s3-007-293847485-724784"},
"X-Amz-Server-Side-Encryption-Context": []string{"{\"bucket\": \"some-bucket\"}"},
}, ShouldFail: false}, // 5
{Header: http.Header{
"X-Amz-Server-Side-Encryption": []string{"AES256"},
"X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id": []string{"s3-007-293847485-724784"},
"X-Amz-Server-Side-Encryption-Context": []string{"{\"bucket\": \"some-bucket\"}"},
}, ShouldFail: true}, // 6
{Header: http.Header{
"X-Amz-Server-Side-Encryption": []string{"aws:kms"},
"X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id": []string{"s3-007-293847485-724784"},
"X-Amz-Server-Side-Encryption-Context": []string{"{\"bucket\": \"some-bucket\""}, // invalid JSON
}, ShouldFail: true}, // 7
}
func TestKMSParseHTTP(t *testing.T) {
for i, test := range kmsParseHTTPTests {
_, _, err := S3KMS.ParseHTTP(test.Header)
if err == nil && test.ShouldFail {
t.Errorf("Test %d: should fail but succeeded", i)
}
if err != nil && !test.ShouldFail {
t.Errorf("Test %d: should pass but failed with: %v", i, err)
}
}
}
var s3IsRequestedTests = []struct {
Header http.Header
Expected bool
+101
View File
@@ -0,0 +1,101 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/madmin"
)
const (
bgHealingUUID = "0000-0000-0000-0000"
)
// NewBgHealSequence creates a background healing sequence
// operation which crawls all objects and heal them.
func newBgHealSequence(numDisks int) *healSequence {
reqInfo := &logger.ReqInfo{API: "BackgroundHeal"}
ctx := logger.SetReqInfo(context.Background(), reqInfo)
hs := madmin.HealOpts{
// Remove objects that do not have read-quorum
Remove: true,
ScanMode: madmin.HealNormalScan,
}
return &healSequence{
sourceCh: make(chan string),
startTime: UTCNow(),
clientToken: bgHealingUUID,
settings: hs,
currentStatus: healSequenceStatus{
Summary: healNotStartedStatus,
HealSettings: hs,
NumDisks: numDisks,
updateLock: &sync.RWMutex{},
},
traverseAndHealDoneCh: make(chan error),
stopSignalCh: make(chan struct{}),
ctx: ctx,
reportProgress: false,
}
}
func getLocalBackgroundHealStatus() madmin.BgHealState {
backgroundSequence, ok := globalSweepHealState.getHealSequenceByToken(bgHealingUUID)
if !ok {
return madmin.BgHealState{}
}
return madmin.BgHealState{
ScannedItemsCount: backgroundSequence.scannedItemsCount,
LastHealActivity: backgroundSequence.lastHealActivity,
}
}
func initDailyHeal() {
go startDailyHeal()
}
func startDailyHeal() {
var objAPI ObjectLayer
var ctx = context.Background()
// Wait until the object API is ready
for {
objAPI = newObjectLayerFn()
if objAPI == nil {
time.Sleep(time.Second)
continue
}
break
}
// Find number of disks in the setup
info := objAPI.StorageInfo(ctx)
numDisks := info.Backend.OnlineDisks + info.Backend.OfflineDisks
nh := newBgHealSequence(numDisks)
globalSweepHealState.LaunchNewHealSequence(nh)
registerDailySweepListener(nh.sourceCh)
}
+145
View File
@@ -0,0 +1,145 @@
/*
* MinIO Cloud Storage, (C) 2019 MinIO, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
)
// The list of modules listening for the daily listing of all objects
// such as the daily heal ops, disk usage and bucket lifecycle management.
var globalDailySweepListeners = make([]chan string, 0)
var globalDailySweepListenersMu = sync.Mutex{}
// Add a new listener to the daily objects listing
func registerDailySweepListener(ch chan string) {
globalDailySweepListenersMu.Lock()
defer globalDailySweepListenersMu.Unlock()
globalDailySweepListeners = append(globalDailySweepListeners, ch)
}
// Safe copy of globalDailySweepListeners content
func copyDailySweepListeners() []chan string {
globalDailySweepListenersMu.Lock()
defer globalDailySweepListenersMu.Unlock()
var listenersCopy = make([]chan string, len(globalDailySweepListeners))
copy(listenersCopy, globalDailySweepListeners)
return listenersCopy
}
// sweepRound will list all objects, having read quorum or not and
// feeds to all listeners, such as the background healing
func sweepRound(ctx context.Context, objAPI ObjectLayer) error {
zeroDuration := time.Millisecond
zeroDynamicTimeout := newDynamicTimeout(zeroDuration, zeroDuration)
// General lock so we avoid parallel daily sweep by different instances.
sweepLock := globalNSMutex.NewNSLock(ctx, "system", "daily-sweep")
if err := sweepLock.GetLock(zeroDynamicTimeout); err != nil {
return err
}
defer sweepLock.Unlock()
buckets, err := objAPI.ListBuckets(ctx)
if err != nil {
return err
}
// List all objects, having read quorum or not in all buckets
// and send them to all the registered sweep listeners
for _, bucket := range buckets {
// Send bucket names to all listeners
for _, l := range copyDailySweepListeners() {
l <- bucket.Name
}
marker := ""
for {
res, err := objAPI.ListObjectsHeal(ctx, bucket.Name, "", marker, "", 1000)
if err != nil {
continue
}
for _, obj := range res.Objects {
for _, l := range copyDailySweepListeners() {
l <- pathJoin(bucket.Name, obj.Name)
}
}
if !res.IsTruncated {
break
} else {
marker = res.NextMarker
}
}
}
return nil
}
// initDailySweeper creates a go-routine which will list all
// objects in all buckets in a daily basis
func initDailySweeper() {
go dailySweeper()
}
// List all objects in all buckets in a daily basis
func dailySweeper() {
var lastSweepTime time.Time
var objAPI ObjectLayer
var ctx = context.Background()
// Wait until the object layer is ready
for {
objAPI = newObjectLayerFn()
if objAPI == nil {
time.Sleep(time.Second)
continue
}
break
}
// Perform a sweep round each 24 hours
for {
if time.Since(lastSweepTime) < 24*time.Hour {
time.Sleep(time.Hour)
continue
}
err := sweepRound(ctx, objAPI)
if err != nil {
switch err.(type) {
// Unable to hold a lock means there is another
// instance doing the sweep round
case OperationTimedOut:
lastSweepTime = time.Now()
default:
logger.LogIf(ctx, err)
time.Sleep(time.Minute)
continue
}
} else {
lastSweepTime = time.Now()
}
}
}
+3 -4
View File
@@ -18,7 +18,6 @@ package cmd
import (
"context"
"encoding/hex"
"encoding/json"
"fmt"
"io"
@@ -304,7 +303,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
data := r.Reader
fs := cfs.FSObjects
// Lock the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetLock(globalObjectTimeout); err != nil {
return objInfo, err
}
@@ -399,7 +398,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
if fsMeta.Meta["etag"] == "" {
fsMeta.Meta["etag"] = hex.EncodeToString(data.MD5Current())
fsMeta.Meta["etag"] = r.MD5CurrentHexString()
}
// Should return IncompleteBody{} error when reader has fewer
// bytes than specified in request header.
@@ -490,7 +489,7 @@ func (cfs *cacheFSObjects) NewMultipartUpload(ctx context.Context, bucket, objec
// moveBucketToTrash clears cacheFSObjects of bucket contents and moves it to trash folder.
func (cfs *cacheFSObjects) moveBucketToTrash(ctx context.Context, bucket string) (err error) {
fs := cfs.FSObjects
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if err = bucketLock.GetLock(globalObjectTimeout); err != nil {
return err
}
+45 -31
View File
@@ -62,6 +62,7 @@ type cacheObjects struct {
GetObjectInfoFn func(ctx context.Context, bucket, object string, opts ObjectOptions) (objInfo ObjectInfo, err error)
PutObjectFn func(ctx context.Context, bucket, object string, data *PutObjReader, opts ObjectOptions) (objInfo ObjectInfo, err error)
DeleteObjectFn func(ctx context.Context, bucket, object string) error
DeleteObjectsFn func(ctx context.Context, bucket string, objects []string) ([]error, error)
ListObjectsFn func(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (result ListObjectsInfo, err error)
ListObjectsV2Fn func(ctx context.Context, bucket, prefix, continuationToken, delimiter string, maxKeys int, fetchOwner bool, startAfter string) (result ListObjectsV2Info, err error)
ListBucketsFn func(ctx context.Context) (buckets []BucketInfo, err error)
@@ -94,6 +95,7 @@ type CacheObjectLayer interface {
GetObjectInfo(ctx context.Context, bucket, object string, opts ObjectOptions) (objInfo ObjectInfo, err error)
PutObject(ctx context.Context, bucket, object string, data *PutObjReader, opts ObjectOptions) (objInfo ObjectInfo, err error)
DeleteObject(ctx context.Context, bucket, object string) error
DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error)
// Multipart operations.
NewMultipartUpload(ctx context.Context, bucket, object string, opts ObjectOptions) (uploadID string, err error)
@@ -200,9 +202,12 @@ func (c cacheObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
return cacheReader, nil
} else if err != nil {
if _, ok := err.(ObjectNotFound); ok {
cacheReader.Close()
// Delete cached entry if backend object was deleted.
dcache.Delete(ctx, bucket, object)
if cacheErr == nil {
cacheReader.Close()
// Delete cached entry if backend object
// was deleted.
dcache.Delete(ctx, bucket, object)
}
}
return nil, err
}
@@ -240,7 +245,7 @@ func (c cacheObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
// Initialize pipe.
pipeReader, pipeWriter := io.Pipe()
teeReader := io.TeeReader(bkReader, pipeWriter)
hashReader, herr := hash.NewReader(pipeReader, bkReader.ObjInfo.Size, "", "", bkReader.ObjInfo.Size)
hashReader, herr := hash.NewReader(pipeReader, bkReader.ObjInfo.Size, "", "", bkReader.ObjInfo.Size, globalCLIContext.StrictS3Compat)
if herr != nil {
bkReader.Close()
return nil, herr
@@ -311,7 +316,7 @@ func (c cacheObjects) GetObject(ctx context.Context, bucket, object string, star
}
// Initialize pipe.
pipeReader, pipeWriter := io.Pipe()
hashReader, err := hash.NewReader(pipeReader, objInfo.Size, "", "", objInfo.Size)
hashReader, err := hash.NewReader(pipeReader, objInfo.Size, "", "", objInfo.Size, globalCLIContext.StrictS3Compat)
if err != nil {
return err
}
@@ -372,7 +377,7 @@ func (c cacheObjects) GetObjectInfo(ctx context.Context, bucket, object string,
// Returns function "listDir" of the type listDirFunc.
// isLeaf - is used by listDir function to check if an entry is a leaf or non-leaf entry.
// disks - list of fsObjects
func listDirCacheFactory(isLeaf IsLeafFunc, disks []*cacheFSObjects) ListDirFunc {
func listDirCacheFactory(isLeaf func(string, string) bool, disks []*cacheFSObjects) ListDirFunc {
listCacheDirs := func(bucket, prefixDir, prefixEntry string) (dirs []string) {
var entries []string
for _, disk := range disks {
@@ -388,6 +393,12 @@ func listDirCacheFactory(isLeaf IsLeafFunc, disks []*cacheFSObjects) ListDirFunc
continue
}
for i := range entries {
if isLeaf(bucket, entries[i]) {
entries[i] = strings.TrimSuffix(entries[i], slashSeparator)
}
}
// Filter entries that have the prefix prefixEntry.
entries = filterMatchingPrefix(entries, prefixEntry)
dirs = append(dirs, entries...)
@@ -396,7 +407,7 @@ func listDirCacheFactory(isLeaf IsLeafFunc, disks []*cacheFSObjects) ListDirFunc
}
// listDir - lists all the entries at a given prefix and given entry in the prefix.
listDir := func(bucket, prefixDir, prefixEntry string) (mergedEntries []string, delayIsLeaf bool) {
listDir := func(bucket, prefixDir, prefixEntry string) (mergedEntries []string) {
cacheEntries := listCacheDirs(bucket, prefixDir, prefixEntry)
for _, entry := range cacheEntries {
// Find elements in entries which are not in mergedEntries
@@ -408,7 +419,7 @@ func listDirCacheFactory(isLeaf IsLeafFunc, disks []*cacheFSObjects) ListDirFunc
mergedEntries = append(mergedEntries, entry)
sort.Strings(mergedEntries)
}
return mergedEntries, false
return mergedEntries
}
return listDir
}
@@ -424,28 +435,19 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
if delimiter == slashSeparator {
recursive = false
}
walkResultCh, endWalkCh := c.listPool.Release(listParams{bucket, recursive, marker, prefix})
walkResultCh, endWalkCh := c.listPool.Release(listParams{bucket, recursive, marker, prefix, false})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := func(bucket, object string) bool {
listDir := listDirCacheFactory(func(bucket, object string) bool {
fs, err := c.cache.getCacheFS(ctx, bucket, object)
if err != nil {
return false
}
_, err = fs.getObjectInfo(ctx, bucket, object)
return err == nil
}
isLeafDir := func(bucket, object string) bool {
fs, err := c.cache.getCacheFS(ctx, bucket, object)
if err != nil {
return false
}
return fs.isObjectDir(bucket, object)
}
listDir := listDirCacheFactory(isLeaf, c.cache.cfs)
walkResultCh = startTreeWalk(ctx, bucket, prefix, marker, recursive, listDir, isLeaf, isLeafDir, endWalkCh)
}, c.cache.cfs)
walkResultCh = startTreeWalk(ctx, bucket, prefix, marker, recursive, listDir, endWalkCh)
}
for i := 0; i < maxKeys; {
@@ -455,10 +457,6 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
eof = true
break
}
// For any walk error return right away.
if walkResult.err != nil {
return result, toObjectErr(walkResult.err, bucket, prefix)
}
entry := walkResult.entry
var objInfo ObjectInfo
@@ -496,7 +494,7 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
}
}
params := listParams{bucket, recursive, nextMarker, prefix}
params := listParams{bucket, recursive, nextMarker, prefix, false}
if !eof {
c.listPool.Set(params, walkResultCh, endWalkCh)
}
@@ -633,6 +631,14 @@ func (c cacheObjects) DeleteObject(ctx context.Context, bucket, object string) (
return
}
func (c cacheObjects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = c.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
// Returns true if object should be excluded from cache
func (c cacheObjects) isCacheExclude(bucket, object string) bool {
for _, pattern := range c.exclude {
@@ -668,13 +674,13 @@ func (c cacheObjects) PutObject(ctx context.Context, bucket, object string, r *P
objInfo = ObjectInfo{}
// Initialize pipe to stream data to backend
pipeReader, pipeWriter := io.Pipe()
hashReader, err := hash.NewReader(pipeReader, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize())
hashReader, err := hash.NewReader(pipeReader, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize(), globalCLIContext.StrictS3Compat)
if err != nil {
return ObjectInfo{}, err
}
// Initialize pipe to stream data to cache
rPipe, wPipe := io.Pipe()
cHashReader, err := hash.NewReader(rPipe, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize())
cHashReader, err := hash.NewReader(rPipe, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize(), globalCLIContext.StrictS3Compat)
if err != nil {
return ObjectInfo{}, err
}
@@ -762,13 +768,13 @@ func (c cacheObjects) PutObjectPart(ctx context.Context, bucket, object, uploadI
info = PartInfo{}
// Initialize pipe to stream data to backend
pipeReader, pipeWriter := io.Pipe()
hashReader, err := hash.NewReader(pipeReader, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize())
hashReader, err := hash.NewReader(pipeReader, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize(), globalCLIContext.StrictS3Compat)
if err != nil {
return
}
// Initialize pipe to stream data to cache
rPipe, wPipe := io.Pipe()
cHashReader, err := hash.NewReader(rPipe, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize())
cHashReader, err := hash.NewReader(rPipe, size, data.MD5HexString(), data.SHA256HexString(), data.ActualSize(), globalCLIContext.StrictS3Compat)
if err != nil {
return
}
@@ -978,6 +984,14 @@ func newServerCacheObjects(config CacheConfig) (CacheObjectLayer, error) {
DeleteObjectFn: func(ctx context.Context, bucket, object string) error {
return newObjectLayerFn().DeleteObject(ctx, bucket, object)
},
DeleteObjectsFn: func(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = newObjectLayerFn().DeleteObject(ctx, bucket, object)
}
return errs, nil
},
ListObjectsFn: func(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (result ListObjectsInfo, err error) {
return newObjectLayerFn().ListObjects(ctx, bucket, prefix, marker, delimiter, maxKeys)
},
+2 -2
View File
@@ -194,7 +194,7 @@ func TestDiskCache(t *testing.T) {
objInfo.UserDefined = httpMeta
var opts ObjectOptions
byteReader := bytes.NewReader([]byte(content))
hashReader, err := hash.NewReader(byteReader, int64(size), "", "", int64(size))
hashReader, err := hash.NewReader(byteReader, int64(size), "", "", int64(size), globalCLIContext.StrictS3Compat)
if err != nil {
t.Fatal(err)
}
@@ -269,7 +269,7 @@ func TestDiskCacheMaxUse(t *testing.T) {
opts := ObjectOptions{}
byteReader := bytes.NewReader([]byte(content))
hashReader, err := hash.NewReader(byteReader, int64(size), "", "", int64(size))
hashReader, err := hash.NewReader(byteReader, int64(size), "", "", int64(size), globalCLIContext.StrictS3Compat)
if err != nil {
t.Fatal(err)
}
+12 -1
View File
@@ -29,7 +29,7 @@ import (
"path"
"strconv"
"github.com/minio/minio-go/pkg/encrypt"
"github.com/minio/minio-go/v6/pkg/encrypt"
"github.com/minio/minio/cmd/crypto"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/ioutil"
@@ -1238,6 +1238,17 @@ func putOpts(ctx context.Context, r *http.Request, bucket, object string, metada
opts.UserDefined = metadata
return
}
if crypto.S3KMS.IsRequested(r.Header) {
keyID, context, err := crypto.S3KMS.ParseHTTP(r.Header)
if err != nil {
return ObjectOptions{}, err
}
sseKms, err := encrypt.NewSSEKMS(keyID, context)
if err != nil {
return ObjectOptions{}, err
}
return ObjectOptions{ServerSideEncryption: sseKms, UserDefined: metadata}, nil
}
// default case of passing encryption headers and UserDefined metadata to backend
return getDefaultOpts(r.Header, false, metadata)
}
+1 -1
View File
@@ -23,7 +23,7 @@ import (
"testing"
humanize "github.com/dustin/go-humanize"
"github.com/minio/minio-go/pkg/encrypt"
"github.com/minio/minio-go/v6/pkg/encrypt"
"github.com/minio/minio/cmd/crypto"
"github.com/minio/sio"
)
+6 -6
View File
@@ -22,7 +22,7 @@ import (
"strconv"
"strings"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio-go/v6/pkg/set"
"github.com/minio/minio/pkg/ellipses"
)
@@ -213,12 +213,12 @@ func parseEndpointSet(args ...string) (ep endpointSet, err error) {
return ep, nil
}
// Parses all ellipses input arguments, expands them into corresponding
// list of endpoints chunked evenly in accordance with a specific
// set size.
// GetAllSets - parses all ellipses input arguments, expands them into
// corresponding list of endpoints chunked evenly in accordance with a
// specific set size.
// For example: {1...64} is divided into 4 sets each of size 16.
// This applies to even distributed setup syntax as well.
func getAllSets(args ...string) ([][]string, error) {
func GetAllSets(args ...string) ([][]string, error) {
if len(args) == 0 {
return nil, errInvalidArgument
}
@@ -266,7 +266,7 @@ func getAllSets(args ...string) ([][]string, error) {
// CreateServerEndpoints - validates and creates new endpoints from input args, supports
// both ellipses and without ellipses transparently.
func createServerEndpoints(serverAddr string, args ...string) (string, EndpointList, SetupType, int, int, error) {
setArgs, err := getAllSets(args...)
setArgs, err := GetAllSets(args...)
if err != nil {
return serverAddr, nil, -1, 0, 0, err
}
-1
View File
@@ -44,7 +44,6 @@ func TestCreateServerEndpoints(t *testing.T) {
{":9000", []string{"/export1{1...32}", "/export1{1...32}"}, false},
// Same host cannot export same disk on two ports - special case localhost.
{":9001", []string{"http://localhost:900{1...2}/export{1...64}"}, false},
// Valid inputs.
{":9000", []string{"/export1"}, true},
{":9000", []string{"/export1", "/export2", "/export3", "/export4"}, true},
+122 -8
View File
@@ -17,6 +17,7 @@
package cmd
import (
"context"
"fmt"
"net"
"net/url"
@@ -26,8 +27,11 @@ import (
"runtime"
"strconv"
"strings"
"time"
"github.com/minio/minio-go/pkg/set"
humanize "github.com/dustin/go-humanize"
"github.com/minio/cli"
"github.com/minio/minio-go/v6/pkg/set"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/cpu"
"github.com/minio/minio/pkg/disk"
@@ -44,6 +48,8 @@ const (
// URLEndpointType - URL style endpoint type enum.
URLEndpointType
retryInterval = 5 // In Seconds.
)
// Endpoint - any type of endpoint.
@@ -51,6 +57,7 @@ type Endpoint struct {
*url.URL
IsLocal bool
SetIndex int
HostName string
}
func (endpoint Endpoint) String() string {
@@ -75,6 +82,19 @@ func (endpoint Endpoint) IsHTTPS() bool {
return endpoint.Scheme == "https"
}
// UpdateIsLocal - resolves the host and updates if it is local or not.
func (endpoint *Endpoint) UpdateIsLocal() error {
if !endpoint.IsLocal {
isLocal, err := isLocalHost(endpoint.HostName)
if err != nil {
return err
}
endpoint.IsLocal = isLocal
}
return nil
}
// NewEndpoint - returns new endpoint based on given arguments.
func NewEndpoint(arg string) (ep Endpoint, e error) {
// isEmptyPath - check whether given path is not empty.
@@ -87,6 +107,7 @@ func NewEndpoint(arg string) (ep Endpoint, e error) {
}
var isLocal bool
var host string
u, err := url.Parse(arg)
if err == nil && u.Host != "" {
// URL style of endpoint.
@@ -98,7 +119,7 @@ func NewEndpoint(arg string) (ep Endpoint, e error) {
return ep, fmt.Errorf("invalid URL endpoint format")
}
var host, port string
var port string
host, port, err = net.SplitHostPort(u.Host)
if err != nil {
if !strings.Contains(err.Error(), "missing port in address") {
@@ -150,10 +171,6 @@ func NewEndpoint(arg string) (ep Endpoint, e error) {
}
}
isLocal, err = isLocalHost(host)
if err != nil {
return ep, err
}
} else {
// Only check if the arg is an ip address and ask for scheme since its absent.
// localhost, example.com, any FQDN cannot be disambiguated from a regular file path such as
@@ -166,8 +183,9 @@ func NewEndpoint(arg string) (ep Endpoint, e error) {
}
return Endpoint{
URL: u,
IsLocal: isLocal,
URL: u,
IsLocal: isLocal,
HostName: host,
}, nil
}
@@ -200,6 +218,74 @@ func (endpoints EndpointList) GetString(i int) string {
return endpoints[i].String()
}
// UpdateIsLocal - resolves the host and discovers the local host.
func (endpoints EndpointList) UpdateIsLocal() error {
var epsResolved int
var foundLocal bool
resolvedList := make([]bool, len(endpoints))
// Mark the starting time
startTime := time.Now()
keepAliveTicker := time.NewTicker(retryInterval * time.Second)
defer keepAliveTicker.Stop()
for {
// Break if the local endpoint is found already. Or all the endpoints are resolved.
if foundLocal || (epsResolved == len(endpoints)) {
break
}
// Retry infinitely on Kubernetes and Docker swarm.
// This is needed as the remote hosts are sometime
// not available immediately.
select {
case <-globalOSSignalCh:
return fmt.Errorf("The endpoint resolution got interrupted")
default:
for i, resolved := range resolvedList {
if resolved {
continue
}
// return err if not Docker or Kubernetes
// We use IsDocker() method to check for Docker Swarm environment
// as there is no reliable way to clearly identify Swarm from
// Docker environment.
isLocal, err := isLocalHost(endpoints[i].HostName)
if err != nil {
if !IsDocker() && !IsKubernetes() {
return err
}
// time elapsed
timeElapsed := time.Since(startTime)
// log error only if more than 1s elapsed
if timeElapsed > time.Second {
// log the message to console about the host not being
// resolveable.
reqInfo := (&logger.ReqInfo{}).AppendTags("host", endpoints[i].HostName)
reqInfo.AppendTags("elapsedTime", humanize.RelTime(startTime, startTime.Add(timeElapsed), "elapsed", ""))
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
} else {
resolvedList[i] = true
endpoints[i].IsLocal = isLocal
epsResolved++
if !foundLocal {
foundLocal = isLocal
}
}
}
// Wait for the tick, if the there exist a local endpoint in discovery.
// Non docker/kubernetes environment does not need to wait.
if !foundLocal && (IsDocker() && IsKubernetes()) {
<-keepAliveTicker.C
}
}
}
return nil
}
// localEndpointsMemUsage - returns ServerMemUsageInfo for only the
// local endpoints from given list of endpoints
func localEndpointsMemUsage(endpoints EndpointList) ServerMemUsageInfo {
@@ -302,9 +388,30 @@ func NewEndpointList(args ...string) (endpoints EndpointList, err error) {
uniqueArgs.Add(arg)
endpoints = append(endpoints, endpoint)
}
return endpoints, nil
}
func checkEndpointsSubOptimal(ctx *cli.Context, setupType SetupType, endpoints EndpointList) (err error) {
// Validate sub optimal ordering only for distributed setup.
if setupType != DistXLSetupType {
return nil
}
var endpointOrder int
err = fmt.Errorf("Too many disk args are local, input is in sub-optimal order. Please review input args: %s", ctx.Args())
for _, endpoint := range endpoints {
if endpoint.IsLocal {
endpointOrder++
} else {
endpointOrder--
}
if endpointOrder >= 2 {
return err
}
}
return nil
}
// Checks if there are any cross device mounts.
func checkCrossDeviceMounts(endpoints EndpointList) (err error) {
var absPaths []string
@@ -341,6 +448,9 @@ func CreateEndpoints(serverAddr string, args ...[]string) (string, EndpointList,
if err != nil {
return serverAddr, endpoints, setupType, err
}
if err := endpoint.UpdateIsLocal(); err != nil {
return serverAddr, endpoints, setupType, err
}
if endpoint.Type() != PathEndpointType {
return serverAddr, endpoints, setupType, uiErrInvalidFSEndpoint(nil).Msg("use path style endpoint for FS setup")
}
@@ -381,6 +491,10 @@ func CreateEndpoints(serverAddr string, args ...[]string) (string, EndpointList,
return serverAddr, endpoints, setupType, nil
}
if err := endpoints.UpdateIsLocal(); err != nil {
return serverAddr, endpoints, setupType, uiErrInvalidErasureEndpoints(nil).Msg(err.Error())
}
// Here all endpoints are URL style.
endpointPathSet := set.NewStringSet()
localEndpointCount := 0
+83 -29
View File
@@ -17,13 +17,53 @@
package cmd
import (
"flag"
"fmt"
"net/url"
"reflect"
"strings"
"testing"
"github.com/minio/cli"
)
func TestSubOptimalEndpointInput(t *testing.T) {
args1 := []string{"http://localhost/d1", "http://localhost/d2", "http://localhost/d3", "http://localhost/d4"}
args2 := []string{"http://example.org/d1", "http://example.com/d1", "http://example.net/d1", "http://example.edu/d1"}
tests := []struct {
setupType SetupType
ctx *cli.Context
endpoints EndpointList
isErr bool
}{
{
setupType: DistXLSetupType,
ctx: cli.NewContext(cli.NewApp(), flag.NewFlagSet("", flag.ContinueOnError), nil),
endpoints: mustGetNewEndpointList(args1...),
isErr: false,
},
{
setupType: DistXLSetupType,
ctx: cli.NewContext(cli.NewApp(), flag.NewFlagSet("", flag.ContinueOnError), nil),
endpoints: mustGetNewEndpointList(args2...),
isErr: false,
},
}
for i, test := range tests {
test := test
t.Run(fmt.Sprintf("Test%d", i+1), func(t *testing.T) {
err := checkEndpointsSubOptimal(test.ctx, test.setupType, test.endpoints)
if test.isErr && err == nil {
t.Error("expected err but found nil")
}
if !test.isErr && err != nil {
t.Errorf("expected err nil but found an err %s", err)
}
})
}
}
func TestNewEndpoint(t *testing.T) {
u1, _ := url.Parse("http://localhost/path")
u2, _ := url.Parse("https://example.org/path")
@@ -49,11 +89,11 @@ func TestNewEndpoint(t *testing.T) {
{"http:path", Endpoint{URL: &url.URL{Path: "http:path"}, IsLocal: true}, PathEndpointType, nil},
{"http:/path", Endpoint{URL: &url.URL{Path: "http:/path"}, IsLocal: true}, PathEndpointType, nil},
{"http:///path", Endpoint{URL: &url.URL{Path: "http:/path"}, IsLocal: true}, PathEndpointType, nil},
{"http://localhost/path", Endpoint{URL: u1, IsLocal: true}, URLEndpointType, nil},
{"http://localhost/path//", Endpoint{URL: u1, IsLocal: true}, URLEndpointType, nil},
{"https://example.org/path", Endpoint{URL: u2}, URLEndpointType, nil},
{"http://127.0.0.1:8080/path", Endpoint{URL: u3, IsLocal: true}, URLEndpointType, nil},
{"http://192.168.253.200/path", Endpoint{URL: u4}, URLEndpointType, nil},
{"http://localhost/path", Endpoint{URL: u1, IsLocal: true, HostName: "localhost"}, URLEndpointType, nil},
{"http://localhost/path//", Endpoint{URL: u1, IsLocal: true, HostName: "localhost"}, URLEndpointType, nil},
{"https://example.org/path", Endpoint{URL: u2, IsLocal: false, HostName: "example.org"}, URLEndpointType, nil},
{"http://127.0.0.1:8080/path", Endpoint{URL: u3, IsLocal: true, HostName: "127.0.0.1"}, URLEndpointType, nil},
{"http://192.168.253.200/path", Endpoint{URL: u4, IsLocal: false, HostName: "192.168.253.200"}, URLEndpointType, nil},
{"", Endpoint{}, -1, fmt.Errorf("empty or root endpoint is not supported")},
{"/", Endpoint{}, -1, fmt.Errorf("empty or root endpoint is not supported")},
{`\`, Endpoint{}, -1, fmt.Errorf("empty or root endpoint is not supported")},
@@ -71,6 +111,10 @@ func TestNewEndpoint(t *testing.T) {
for _, testCase := range testCases {
endpoint, err := NewEndpoint(testCase.arg)
if err == nil {
err = endpoint.UpdateIsLocal()
}
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
@@ -261,46 +305,46 @@ func TestCreateEndpoints(t *testing.T) {
// DistXL type
{"127.0.0.1:10000", [][]string{{case1Endpoint1, case1Endpoint2, "http://example.org/d3", "http://example.com/d4"}}, "127.0.0.1:10000", EndpointList{
Endpoint{URL: case1URLs[0], IsLocal: case1LocalFlags[0]},
Endpoint{URL: case1URLs[1], IsLocal: case1LocalFlags[1]},
Endpoint{URL: case1URLs[2], IsLocal: case1LocalFlags[2]},
Endpoint{URL: case1URLs[3], IsLocal: case1LocalFlags[3]},
Endpoint{URL: case1URLs[0], IsLocal: case1LocalFlags[0], HostName: nonLoopBackIP},
Endpoint{URL: case1URLs[1], IsLocal: case1LocalFlags[1], HostName: nonLoopBackIP},
Endpoint{URL: case1URLs[2], IsLocal: case1LocalFlags[2], HostName: "example.org"},
Endpoint{URL: case1URLs[3], IsLocal: case1LocalFlags[3], HostName: "example.com"},
}, DistXLSetupType, nil},
{"127.0.0.1:10000", [][]string{{case2Endpoint1, case2Endpoint2, "http://example.org/d3", "http://example.com/d4"}}, "127.0.0.1:10000", EndpointList{
Endpoint{URL: case2URLs[0], IsLocal: case2LocalFlags[0]},
Endpoint{URL: case2URLs[1], IsLocal: case2LocalFlags[1]},
Endpoint{URL: case2URLs[2], IsLocal: case2LocalFlags[2]},
Endpoint{URL: case2URLs[3], IsLocal: case2LocalFlags[3]},
Endpoint{URL: case2URLs[0], IsLocal: case2LocalFlags[0], HostName: nonLoopBackIP},
Endpoint{URL: case2URLs[1], IsLocal: case2LocalFlags[1], HostName: nonLoopBackIP},
Endpoint{URL: case2URLs[2], IsLocal: case2LocalFlags[2], HostName: "example.org"},
Endpoint{URL: case2URLs[3], IsLocal: case2LocalFlags[3], HostName: "example.com"},
}, DistXLSetupType, nil},
{":80", [][]string{{case3Endpoint1, "http://example.org:9000/d2", "http://example.com/d3", "http://example.net/d4"}}, ":80", EndpointList{
Endpoint{URL: case3URLs[0], IsLocal: case3LocalFlags[0]},
Endpoint{URL: case3URLs[1], IsLocal: case3LocalFlags[1]},
Endpoint{URL: case3URLs[2], IsLocal: case3LocalFlags[2]},
Endpoint{URL: case3URLs[3], IsLocal: case3LocalFlags[3]},
Endpoint{URL: case3URLs[0], IsLocal: case3LocalFlags[0], HostName: nonLoopBackIP},
Endpoint{URL: case3URLs[1], IsLocal: case3LocalFlags[1], HostName: "example.org"},
Endpoint{URL: case3URLs[2], IsLocal: case3LocalFlags[2], HostName: "example.com"},
Endpoint{URL: case3URLs[3], IsLocal: case3LocalFlags[3], HostName: "example.net"},
}, DistXLSetupType, nil},
{":9000", [][]string{{case4Endpoint1, "http://example.org/d2", "http://example.com/d3", "http://example.net/d4"}}, ":9000", EndpointList{
Endpoint{URL: case4URLs[0], IsLocal: case4LocalFlags[0]},
Endpoint{URL: case4URLs[1], IsLocal: case4LocalFlags[1]},
Endpoint{URL: case4URLs[2], IsLocal: case4LocalFlags[2]},
Endpoint{URL: case4URLs[3], IsLocal: case4LocalFlags[3]},
Endpoint{URL: case4URLs[0], IsLocal: case4LocalFlags[0], HostName: nonLoopBackIP},
Endpoint{URL: case4URLs[1], IsLocal: case4LocalFlags[1], HostName: "example.org"},
Endpoint{URL: case4URLs[2], IsLocal: case4LocalFlags[2], HostName: "example.com"},
Endpoint{URL: case4URLs[3], IsLocal: case4LocalFlags[3], HostName: "example.net"},
}, DistXLSetupType, nil},
{":9000", [][]string{{case5Endpoint1, case5Endpoint2, case5Endpoint3, case5Endpoint4}}, ":9000", EndpointList{
Endpoint{URL: case5URLs[0], IsLocal: case5LocalFlags[0]},
Endpoint{URL: case5URLs[1], IsLocal: case5LocalFlags[1]},
Endpoint{URL: case5URLs[2], IsLocal: case5LocalFlags[2]},
Endpoint{URL: case5URLs[3], IsLocal: case5LocalFlags[3]},
Endpoint{URL: case5URLs[0], IsLocal: case5LocalFlags[0], HostName: nonLoopBackIP},
Endpoint{URL: case5URLs[1], IsLocal: case5LocalFlags[1], HostName: nonLoopBackIP},
Endpoint{URL: case5URLs[2], IsLocal: case5LocalFlags[2], HostName: nonLoopBackIP},
Endpoint{URL: case5URLs[3], IsLocal: case5LocalFlags[3], HostName: nonLoopBackIP},
}, DistXLSetupType, nil},
// DistXL Setup using only local host.
{":9003", [][]string{{"http://localhost:9000/d1", "http://localhost:9001/d2", "http://127.0.0.1:9002/d3", case6Endpoint}}, ":9003", EndpointList{
Endpoint{URL: case6URLs[0], IsLocal: case6LocalFlags[0]},
Endpoint{URL: case6URLs[1], IsLocal: case6LocalFlags[1]},
Endpoint{URL: case6URLs[2], IsLocal: case6LocalFlags[2]},
Endpoint{URL: case6URLs[3], IsLocal: case6LocalFlags[3]},
Endpoint{URL: case6URLs[0], IsLocal: case6LocalFlags[0], HostName: "localhost"},
Endpoint{URL: case6URLs[1], IsLocal: case6LocalFlags[1], HostName: "localhost"},
Endpoint{URL: case6URLs[2], IsLocal: case6LocalFlags[2], HostName: "127.0.0.1"},
Endpoint{URL: case6URLs[3], IsLocal: case6LocalFlags[3], HostName: nonLoopBackIP},
}, DistXLSetupType, nil},
}
@@ -357,6 +401,11 @@ func TestGetLocalPeer(t *testing.T) {
for i, testCase := range testCases {
endpoints, _ := NewEndpointList(testCase.endpointArgs...)
if !endpoints[0].IsLocal {
if err := endpoints.UpdateIsLocal(); err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
}
remotePeer := GetLocalPeer(endpoints)
if remotePeer != testCase.expectedResult {
t.Fatalf("Test %d: expected: %v, got: %v", i+1, testCase.expectedResult, remotePeer)
@@ -384,6 +433,11 @@ func TestGetRemotePeers(t *testing.T) {
for _, testCase := range testCases {
endpoints, _ := NewEndpointList(testCase.endpointArgs...)
if !endpoints[0].IsLocal {
if err := endpoints.UpdateIsLocal(); err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
}
remotePeers := GetRemotePeers(endpoints)
if !reflect.DeepEqual(remotePeers, testCase.expectedResult) {
t.Fatalf("expected: %v, got: %v", testCase.expectedResult, remotePeers)
+4 -4
View File
@@ -77,10 +77,7 @@ func (e *Erasure) DecodeDataBlocks(data [][]byte) error {
if !needsReconstruction {
return nil
}
if err := e.encoder.ReconstructData(data); err != nil {
return err
}
return nil
return e.encoder.ReconstructData(data)
}
// DecodeDataAndParityBlocks decodes the given erasure-coded data and verifies it.
@@ -103,6 +100,9 @@ func (e *Erasure) ShardFileSize(totalLength int64) int64 {
if totalLength == 0 {
return 0
}
if totalLength == -1 {
return -1
}
numShards := totalLength / e.blockSize
lastBlockSize := totalLength % int64(e.blockSize)
lastShardSize := ceilFrac(lastBlockSize, int64(e.dataBlocks))
+1 -1
View File
@@ -491,7 +491,7 @@ func formatXLGetDeploymentID(refFormat *formatXLV3, formats []*formatXLV3) (stri
func formatXLFixDeploymentID(ctx context.Context, endpoints EndpointList, storageDisks []StorageAPI, refFormat *formatXLV3) (err error) {
// Acquire lock on format.json
mutex := newNSLock(globalIsDistXL)
formatLock := mutex.NewNSLock(minioMetaBucket, formatConfigFile)
formatLock := mutex.NewNSLock(ctx, minioMetaBucket, formatConfigFile)
if err = formatLock.GetLock(globalHealingTimeout); err != nil {
return err
}
+3
View File
@@ -206,6 +206,9 @@ func osErrToFSFileErr(err error) error {
if isSysErrPathNotFound(err) {
return errFileNotFound
}
if isSysErrTooManyFiles(err) {
return errTooManyOpenFiles
}
return err
}
+2 -5
View File
@@ -504,10 +504,7 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
}
// Calculate s3 compatible md5sum for complete multipart.
s3MD5, err := getCompleteMultipartMD5(ctx, parts)
if err != nil {
return oi, err
}
s3MD5 := getCompleteMultipartMD5(parts)
partSize := int64(-1) // Used later to ensure that all parts sizes are same.
@@ -637,7 +634,7 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
}
// Hold write lock on the object.
destLock := fs.nsMutex.NewNSLock(bucket, object)
destLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err = destLock.GetLock(globalObjectTimeout); err != nil {
return oi, err
}
+43 -41
View File
@@ -30,7 +30,7 @@ import (
"sync/atomic"
"time"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/minio-go/v6/pkg/s3utils"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/lock"
"github.com/minio/minio/pkg/madmin"
@@ -286,7 +286,7 @@ func (fs *FSObjects) statBucketDir(ctx context.Context, bucket string) (os.FileI
// MakeBucketWithLocation - create a new bucket, returns if it
// already exists.
func (fs *FSObjects) MakeBucketWithLocation(ctx context.Context, bucket, location string) error {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if err := bucketLock.GetLock(globalObjectTimeout); err != nil {
return err
}
@@ -309,7 +309,7 @@ func (fs *FSObjects) MakeBucketWithLocation(ctx context.Context, bucket, locatio
// GetBucketInfo - fetch bucket metadata info.
func (fs *FSObjects) GetBucketInfo(ctx context.Context, bucket string) (bi BucketInfo, e error) {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if e := bucketLock.GetRLock(globalObjectTimeout); e != nil {
return bi, e
}
@@ -372,7 +372,7 @@ func (fs *FSObjects) ListBuckets(ctx context.Context) ([]BucketInfo, error) {
// DeleteBucket - delete a bucket and all the metadata associated
// with the bucket including pending multipart, object metadata.
func (fs *FSObjects) DeleteBucket(ctx context.Context, bucket string) error {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
bucketLock := fs.nsMutex.NewNSLock(ctx, bucket, "")
if err := bucketLock.GetLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return err
@@ -408,7 +408,7 @@ func (fs *FSObjects) DeleteBucket(ctx context.Context, bucket string) error {
func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBucket, dstObject string, srcInfo ObjectInfo, srcOpts, dstOpts ObjectOptions) (oi ObjectInfo, e error) {
cpSrcDstSame := isStringEqual(pathJoin(srcBucket, srcObject), pathJoin(dstBucket, dstObject))
if !cpSrcDstSame {
objectDWLock := fs.nsMutex.NewNSLock(dstBucket, dstObject)
objectDWLock := fs.nsMutex.NewNSLock(ctx, dstBucket, dstObject)
if err := objectDWLock.GetLock(globalObjectTimeout); err != nil {
return oi, err
}
@@ -431,8 +431,9 @@ func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
// Save objects' metadata in `fs.json`.
fsMeta := newFSMetaV1()
if _, err = fsMeta.ReadFrom(ctx, wlk); err != nil && err != io.EOF {
return oi, toObjectErr(err, srcBucket, srcObject)
if _, err = fsMeta.ReadFrom(ctx, wlk); err != nil {
// For any error to read fsMeta, set default ETag and proceed.
fsMeta = fs.defaultFsJSON(srcObject)
}
fsMeta.Meta = srcInfo.UserDefined
@@ -450,6 +451,11 @@ func (fs *FSObjects) CopyObject(ctx context.Context, srcBucket, srcObject, dstBu
// Return the new object info.
return fsMeta.ToObjectInfo(srcBucket, srcObject, fi), nil
}
if err := checkPutObjectArgs(ctx, dstBucket, dstObject, fs, srcInfo.PutObjReader.Size()); err != nil {
return ObjectInfo{}, err
}
objInfo, err := fs.putObject(ctx, dstBucket, dstObject, srcInfo.PutObjReader, ObjectOptions{ServerSideEncryption: dstOpts.ServerSideEncryption, UserDefined: srcInfo.UserDefined})
if err != nil {
return oi, toObjectErr(err, dstBucket, dstObject)
@@ -474,7 +480,7 @@ func (fs *FSObjects) GetObjectNInfo(ctx context.Context, bucket, object string,
if lockType != noLock {
// Lock the object before reading.
lock := fs.nsMutex.NewNSLock(bucket, object)
lock := fs.nsMutex.NewNSLock(ctx, bucket, object)
switch lockType {
case writeLock:
if err = lock.GetLock(globalObjectTimeout); err != nil {
@@ -561,7 +567,7 @@ func (fs *FSObjects) GetObject(ctx context.Context, bucket, object string, offse
}
// Lock the object before reading.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetRLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return err
@@ -705,10 +711,7 @@ func (fs *FSObjects) getObjectInfo(ctx context.Context, bucket, object string) (
_, rerr := fsMeta.ReadFrom(ctx, rlk.LockedFile)
fs.rwPool.Close(fsMetaPath)
if rerr != nil {
if rerr != io.EOF {
return oi, rerr
}
// Set Default ETag, if fs.json is empty
// For any error to read fsMeta, set default ETag and proceed.
fsMeta = fs.defaultFsJSON(object)
}
}
@@ -736,7 +739,7 @@ func (fs *FSObjects) getObjectInfo(ctx context.Context, bucket, object string) (
// getObjectInfoWithLock - reads object metadata and replies back ObjectInfo.
func (fs *FSObjects) getObjectInfoWithLock(ctx context.Context, bucket, object string) (oi ObjectInfo, e error) {
// Lock the object before reading.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetRLock(globalObjectTimeout); err != nil {
return oi, err
}
@@ -761,7 +764,7 @@ func (fs *FSObjects) getObjectInfoWithLock(ctx context.Context, bucket, object s
func (fs *FSObjects) GetObjectInfo(ctx context.Context, bucket, object string, opts ObjectOptions) (oi ObjectInfo, e error) {
oi, err := fs.getObjectInfoWithLock(ctx, bucket, object)
if err == errCorruptedFormat || err == io.EOF {
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err = objectLock.GetLock(globalObjectTimeout); err != nil {
return oi, toObjectErr(err, bucket, object)
}
@@ -807,12 +810,13 @@ func (fs *FSObjects) PutObject(ctx context.Context, bucket string, object string
return ObjectInfo{}, err
}
// Lock the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetLock(globalObjectTimeout); err != nil {
logger.LogIf(ctx, err)
return objInfo, err
}
defer objectLock.Unlock()
return fs.putObject(ctx, bucket, object, r, opts)
}
@@ -854,10 +858,6 @@ func (fs *FSObjects) putObject(ctx context.Context, bucket string, object string
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
if err = checkPutObjectArgs(ctx, bucket, object, fs, data.Size()); err != nil {
return ObjectInfo{}, err
}
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
@@ -951,11 +951,21 @@ func (fs *FSObjects) putObject(ctx context.Context, bucket string, object string
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
// DeleteObjects - deletes an object from a bucket, this operation is destructive
// and there are no rollbacks supported.
func (fs *FSObjects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = fs.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
// DeleteObject - deletes an object from a bucket, this operation is destructive
// and there are no rollbacks supported.
func (fs *FSObjects) DeleteObject(ctx context.Context, bucket, object string) error {
// Acquire a write lock before deleting the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
objectLock := fs.nsMutex.NewNSLock(ctx, bucket, object)
if err := objectLock.GetLock(globalOperationTimeout); err != nil {
return err
}
@@ -1001,17 +1011,17 @@ func (fs *FSObjects) DeleteObject(ctx context.Context, bucket, object string) er
// Returns function "listDir" of the type listDirFunc.
// isLeaf - is used by listDir function to check if an entry
// is a leaf or non-leaf entry.
func (fs *FSObjects) listDirFactory(isLeaf IsLeafFunc) ListDirFunc {
func (fs *FSObjects) listDirFactory() ListDirFunc {
// listDir - lists all the entries at a given prefix and given entry in the prefix.
listDir := func(bucket, prefixDir, prefixEntry string) (entries []string, delayIsLeaf bool) {
listDir := func(bucket, prefixDir, prefixEntry string) (entries []string) {
var err error
entries, err = readDir(pathJoin(fs.fsPath, bucket, prefixDir))
if err != nil && err != errFileNotFound {
logger.LogIf(context.Background(), err)
return
}
entries, delayIsLeaf = filterListEntries(bucket, prefixDir, entries, prefixEntry, isLeaf)
return entries, delayIsLeaf
sort.Strings(entries)
return filterMatchingPrefix(entries, prefixEntry)
}
// Return list factory instance.
@@ -1097,22 +1107,8 @@ func (fs *FSObjects) getObjectETag(ctx context.Context, bucket, entry string, lo
// ListObjects - list all objects at prefix upto maxKeys., optionally delimited by '/'. Maintains the list pool
// state for future re-entrant list requests.
func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
isLeaf := func(bucket, object string) bool {
// bucket argument is unused as we don't need to StatFile
// to figure if it's a file, just need to check that the
// object string does not end with "/".
return !hasSuffix(object, slashSeparator)
}
// Return true if the specified object is an empty directory
isLeafDir := func(bucket, object string) bool {
if !hasSuffix(object, slashSeparator) {
return false
}
return fs.isObjectDir(bucket, object)
}
listDir := fs.listDirFactory(isLeaf)
return listObjects(ctx, fs, bucket, prefix, marker, delimiter, maxKeys, fs.listPool, isLeaf, isLeafDir, listDir, fs.getObjectInfo, fs.getObjectInfo)
return listObjects(ctx, fs, bucket, prefix, marker, delimiter, maxKeys, fs.listPool,
fs.listDirFactory(), fs.getObjectInfo, fs.getObjectInfo)
}
// ReloadFormat - no-op for fs, Valid only for XL.
@@ -1153,6 +1149,12 @@ func (fs *FSObjects) ListBucketsHeal(ctx context.Context) ([]BucketInfo, error)
return []BucketInfo{}, NotImplemented{}
}
// ListObjectsHeal - list all objects to be healed. Valid only for XL
func (fs *FSObjects) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (result ListObjectsInfo, err error) {
logger.LogIf(ctx, NotImplemented{})
return ListObjectsInfo{}, NotImplemented{}
}
// SetBucketPolicy sets policy on bucket
func (fs *FSObjects) SetBucketPolicy(ctx context.Context, bucket string, policy *policy.Policy) error {
return savePolicyConfig(ctx, fs, bucket, policy)
+11 -8
View File
@@ -21,10 +21,11 @@ import (
"os"
"strings"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/hash"
minio "github.com/minio/minio-go"
minio "github.com/minio/minio-go/v6"
)
var (
@@ -43,14 +44,11 @@ var (
// ListObjects function alias.
ListObjects = listObjects
// FilterListEntries function alias.
FilterListEntries = filterListEntries
// FilterMatchingPrefix function alias.
FilterMatchingPrefix = filterMatchingPrefix
// IsStringEqual is string equal.
IsStringEqual = isStringEqual
// GetCompleteMultipartMD5 returns multipart MD5
GetCompleteMultipartMD5 = getCompleteMultipartMD5
)
// StatInfo - alias for statInfo
@@ -169,7 +167,7 @@ func FromMinioClientListMultipartsInfo(lmur minio.ListMultipartUploadsResult) Li
// FromMinioClientObjectInfo converts minio ObjectInfo to gateway ObjectInfo
func FromMinioClientObjectInfo(bucket string, oi minio.ObjectInfo) ObjectInfo {
userDefined := FromMinioClientMetadata(oi.Metadata)
userDefined["Content-Type"] = oi.ContentType
userDefined[xhttp.ContentType] = oi.ContentType
return ObjectInfo{
Bucket: bucket,
@@ -179,7 +177,7 @@ func FromMinioClientObjectInfo(bucket string, oi minio.ObjectInfo) ObjectInfo {
ETag: canonicalizeETag(oi.ETag),
UserDefined: userDefined,
ContentType: oi.ContentType,
ContentEncoding: oi.Metadata.Get("Content-Encoding"),
ContentEncoding: oi.Metadata.Get(xhttp.ContentEncoding),
StorageClass: oi.StorageClass,
Expires: oi.Expires,
}
@@ -349,6 +347,11 @@ func ErrorRespToObjectError(err error, params ...string) error {
return err
}
// ComputeCompleteMultipartMD5 calculates MD5 ETag for complete multipart responses
func ComputeCompleteMultipartMD5(parts []CompletePart) string {
return getCompleteMultipartMD5(parts)
}
// parse gateway sse env variable
func parseGatewaySSE(s string) (gatewaySSE, error) {
l := strings.Split(s, ";")
+14 -4
View File
@@ -41,14 +41,14 @@ var (
gatewayCmd = cli.Command{
Name: "gateway",
Usage: "start object storage gateway",
Flags: append(serverFlags, globalFlags...),
Flags: append(ServerFlags, GlobalFlags...),
HideHelpCommand: true,
}
)
// RegisterGatewayCommand registers a new command for gateway.
func RegisterGatewayCommand(cmd cli.Command) error {
cmd.Flags = append(append(cmd.Flags, append(cmd.Flags, serverFlags...)...), globalFlags...)
cmd.Flags = append(append(cmd.Flags, ServerFlags...), GlobalFlags...)
gatewayCmd.Subcommands = append(gatewayCmd.Subcommands, cmd)
return nil
}
@@ -121,7 +121,7 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// to IPv6 address ie minio will start listening on IPv6 address whereas another
// (non-)minio process is listening on IPv4 of given port.
// To avoid this error situation we check for port availability.
logger.FatalIf(checkPortAvailability(globalMinioPort), "Unable to start the gateway")
logger.FatalIf(checkPortAvailability(globalMinioHost, globalMinioPort), "Unable to start the gateway")
// Check and load TLS certificates.
var err error
@@ -148,6 +148,9 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
initNSLock(false) // Enable local namespace lock.
// Set when gateway is enabled
globalIsGateway = true
router := mux.NewRouter().SkipClean(true)
if globalEtcdClient != nil {
@@ -175,9 +178,10 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// Currently only NAS and S3 gateway support encryption headers.
encryptionEnabled := gatewayName == "s3" || gatewayName == "nas"
allowSSEKMS := gatewayName == "s3" // Only S3 can support SSE-KMS (as pass-through)
// Add API router.
registerAPIRouter(router, encryptionEnabled)
registerAPIRouter(router, encryptionEnabled, allowSSEKMS)
var getCert certs.GetCertificateFunc
if globalTLSCerts != nil {
@@ -221,6 +225,11 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
logger.FatalIf(err, "Unable to initialize gateway backend")
}
// Populate existing buckets to the etcd backend
if globalDNSConfig != nil {
initFederatorBackend(newObject)
}
if enableConfigOps {
// Create a new config system.
globalConfigSys = NewConfigSys()
@@ -234,6 +243,7 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
// This is only to uniquely identify each gateway deployments.
globalDeploymentID = os.Getenv("MINIO_GATEWAY_DEPLOYMENT_ID")
logger.SetDeploymentID(globalDeploymentID)
var cacheConfig = globalServerConfig.GetCacheConfig()
if len(cacheConfig.Drives) > 0 {
+31
View File
@@ -17,6 +17,7 @@
package cmd
import (
"fmt"
"strings"
"testing"
@@ -34,6 +35,36 @@ func TestRegisterGatewayCommand(t *testing.T) {
}
}
// Test running a registered gateway command with a flag
func TestRunRegisteredGatewayCommand(t *testing.T) {
var err error
flagName := "test-flag"
flagValue := "foo"
cmd := cli.Command{
Name: "test-run-with-flag",
Flags: []cli.Flag{
cli.StringFlag{Name: flagName},
},
Action: func(ctx *cli.Context) {
if actual := ctx.String(flagName); actual != flagValue {
t.Errorf("value of %s expects %s, but got %s", flagName, flagValue, actual)
}
},
}
err = RegisterGatewayCommand(cmd)
if err != nil {
t.Errorf("RegisterGatewayCommand got unexpected error: %s", err)
}
if err = newApp("minio").Run(
[]string{"minio", "gateway", cmd.Name, fmt.Sprintf("--%s", flagName), flagValue}); err != nil {
t.Errorf("running registered gateway command got unexpected error: %s", err)
}
}
// Test parseGatewayEndpoint
func TestParseGatewayEndpoint(t *testing.T) {
testCases := []struct {
+5
View File
@@ -101,6 +101,11 @@ func (a GatewayUnsupported) ListBucketsHeal(ctx context.Context) (buckets []Buck
return nil, NotImplemented{}
}
// ListObjectsHeal - Not implemented stub
func (a GatewayUnsupported) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (result ListObjectsInfo, err error) {
return ListObjectsInfo{}, NotImplemented{}
}
// HealObject - Not implemented stub
func (a GatewayUnsupported) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (h madmin.HealResultItem, e error) {
return h, NotImplemented{}
+36 -28
View File
@@ -36,7 +36,8 @@ import (
"github.com/Azure/go-autorest/autorest/azure"
humanize "github.com/dustin/go-humanize"
"github.com/minio/cli"
miniogopolicy "github.com/minio/minio-go/pkg/policy"
miniogopolicy "github.com/minio/minio-go/v6/pkg/policy"
"github.com/minio/minio/cmd"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/policy"
@@ -89,23 +90,23 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for Azure Blob Storage backend.
$ export MINIO_ACCESS_KEY=azureaccountname
$ export MINIO_SECRET_KEY=azureaccountkey
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}azureaccountname
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}azureaccountkey
{{.Prompt}} {{.HelpName}}
2. Start minio gateway server for Azure Blob Storage backend on custom endpoint.
$ export MINIO_ACCESS_KEY=azureaccountname
$ export MINIO_SECRET_KEY=azureaccountkey
$ {{.HelpName}} https://azureaccountname.blob.custom.azure.endpoint
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}azureaccountname
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}azureaccountkey
{{.Prompt}} {{.HelpName}} https://azureaccountname.blob.custom.azure.endpoint
3. Start minio gateway server for Azure Blob Storage backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=azureaccountname
$ export MINIO_SECRET_KEY=azureaccountkey
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}azureaccountname
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}azureaccountkey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXPIRY{{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_MAXUSE{{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}}
`
minio.RegisterGatewayCommand(cli.Command{
@@ -834,7 +835,7 @@ func (a *azureObjects) PutObject(ctx context.Context, bucket, object string, r *
}
// Save md5sum for future processing on the object.
opts.UserDefined["x-amz-meta-md5sum"] = hex.EncodeToString(data.MD5Current())
opts.UserDefined["x-amz-meta-md5sum"] = r.MD5CurrentHexString()
objBlob.Metadata, objBlob.Properties, err = s3MetaToAzureProperties(ctx, opts.UserDefined)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
@@ -895,6 +896,14 @@ func (a *azureObjects) DeleteObject(ctx context.Context, bucket, object string)
return nil
}
func (a *azureObjects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = a.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
// ListMultipartUploads - It's decided not to support List Multipart Uploads, hence returning empty result.
func (a *azureObjects) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result minio.ListMultipartsInfo, err error) {
// It's decided not to support List Multipart Uploads, hence returning empty result.
@@ -1202,19 +1211,18 @@ func (a *azureObjects) CompleteMultipartUpload(ctx context.Context, bucket, obje
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
if len(metadata.Metadata) > 0 {
objBlob.Metadata, objBlob.Properties, err = s3MetaToAzureProperties(ctx, metadata.Metadata)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
err = objBlob.SetProperties(nil)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
err = objBlob.SetMetadata(nil)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
objBlob.Metadata, objBlob.Properties, err = s3MetaToAzureProperties(ctx, metadata.Metadata)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
objBlob.Metadata["md5sum"] = cmd.ComputeCompleteMultipartMD5(uploadedParts)
err = objBlob.SetProperties(nil)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
err = objBlob.SetMetadata(nil)
if err != nil {
return objInfo, azureToObjectError(err, bucket, object)
}
var partNumberMarker int
for {
+19 -11
View File
@@ -31,7 +31,7 @@ import (
b2 "github.com/minio/blazer/base"
"github.com/minio/cli"
miniogopolicy "github.com/minio/minio-go/pkg/policy"
miniogopolicy "github.com/minio/minio-go/v6/pkg/policy"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
h2 "github.com/minio/minio/pkg/hash"
@@ -77,18 +77,18 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for B2 backend.
$ export MINIO_ACCESS_KEY=accountID
$ export MINIO_SECRET_KEY=applicationKey
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accountID
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}applicationKey
{{.Prompt}} {{.HelpName}}
2. Start minio gateway server for B2 backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=accountID
$ export MINIO_SECRET_KEY=applicationKey
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accountID
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}applicationKey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXPIRY{{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_MAXUSE{{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}}
`
minio.RegisterGatewayCommand(cli.Command{
Name: b2Backend,
@@ -595,6 +595,14 @@ func (l *b2Objects) DeleteObject(ctx context.Context, bucket string, object stri
return b2ToObjectError(err, bucket, object)
}
func (l *b2Objects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = l.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
// ListMultipartUploads lists all multipart uploads.
func (l *b2Objects) ListMultipartUploads(ctx context.Context, bucket string, prefix string, keyMarker string, uploadIDMarker string,
delimiter string, maxUploads int) (lmi minio.ListMultipartsInfo, err error) {
+34 -25
View File
@@ -19,6 +19,7 @@ package gcs
import (
"context"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
@@ -37,7 +38,7 @@ import (
"cloud.google.com/go/storage"
humanize "github.com/dustin/go-humanize"
"github.com/minio/cli"
miniogopolicy "github.com/minio/minio-go/pkg/policy"
miniogopolicy "github.com/minio/minio-go/v6/pkg/policy"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/policy"
@@ -128,21 +129,21 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for GCS backend.
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
{{.Prompt}} {{.EnvVarSetCommand}} GOOGLE_APPLICATION_CREDENTIALS{{.AssignmentOperator}}/path/to/credentials.json
(Instructions to generate credentials : https://developers.google.com/identity/protocols/application-default-credentials)
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ {{.HelpName}} mygcsprojectid
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}} mygcsprojectid
2. Start minio gateway server for GCS backend with edge caching enabled.
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ {{.HelpName}} mygcsprojectid
{{.Prompt}} {{.EnvVarSetCommand}} GOOGLE_APPLICATION_CREDENTIALS{{.AssignmentOperator}}/path/to/credentials.json
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXPIRY{{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_MAXUSE{{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}} mygcsprojectid
`
minio.RegisterGatewayCommand(cli.Command{
@@ -394,10 +395,6 @@ func (l *gcsGateway) CleanupGCSMinioSysTmp(ctx context.Context) {
for {
attrs, err := it.Next()
if err != nil {
if err != iterator.Done {
ctx := logger.SetReqInfo(context.Background(), &logger.ReqInfo{})
logger.LogIf(ctx, err)
}
break
}
l.CleanupGCSMinioSysTmpBucket(ctx, attrs.Name)
@@ -460,7 +457,6 @@ func (l *gcsGateway) ListBuckets(ctx context.Context) (buckets []minio.BucketInf
}
if ierr != nil {
logger.LogIf(ctx, ierr)
return buckets, gcsToObjectError(ierr)
}
@@ -830,12 +826,17 @@ func fromGCSAttrsToObjectInfo(attrs *storage.ObjectAttrs) minio.ObjectInfo {
if attrs.ContentLanguage != "" {
metadata["Content-Language"] = attrs.ContentLanguage
}
etag := hex.EncodeToString(attrs.MD5)
if etag == "" {
etag = minio.ToS3ETag(fmt.Sprintf("%d", attrs.CRC32C))
}
return minio.ObjectInfo{
Name: attrs.Name,
Bucket: attrs.Bucket,
ModTime: attrs.Updated,
Size: attrs.Size,
ETag: minio.ToS3ETag(fmt.Sprintf("%d", attrs.CRC32C)),
ETag: etag,
UserDefined: metadata,
ContentType: attrs.ContentType,
ContentEncoding: attrs.ContentEncoding,
@@ -915,15 +916,12 @@ func (l *gcsGateway) PutObject(ctx context.Context, bucket string, key string, r
}
// Close the object writer upon success.
w.Close()
attrs, err := object.Attrs(ctx)
if err != nil {
if err := w.Close(); err != nil {
logger.LogIf(ctx, err)
return minio.ObjectInfo{}, gcsToObjectError(err, bucket, key)
}
return fromGCSAttrsToObjectInfo(attrs), nil
return fromGCSAttrsToObjectInfo(w.Attrs()), nil
}
// CopyObject - Copies a blob from source container to destination container.
@@ -958,6 +956,14 @@ func (l *gcsGateway) DeleteObject(ctx context.Context, bucket string, object str
return nil
}
func (l *gcsGateway) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = l.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
// NewMultipartUpload - upload object in multiple parts
func (l *gcsGateway) NewMultipartUpload(ctx context.Context, bucket string, key string, o minio.ObjectOptions) (uploadID string, err error) {
// generate new uploadid
@@ -1093,7 +1099,10 @@ func (l *gcsGateway) PutObjectPart(ctx context.Context, bucket string, key strin
return minio.PartInfo{}, gcsToObjectError(err, bucket, key)
}
// Make sure to close the object writer upon success.
w.Close()
if err := w.Close(); err != nil {
logger.LogIf(ctx, err)
return minio.PartInfo{}, gcsToObjectError(err, bucket, key)
}
return minio.PartInfo{
PartNumber: partNumber,
ETag: etag,
+1 -1
View File
@@ -28,7 +28,7 @@ import (
"cloud.google.com/go/storage"
"google.golang.org/api/googleapi"
miniogo "github.com/minio/minio-go"
miniogo "github.com/minio/minio-go/v6"
minio "github.com/minio/minio/cmd"
)
+1 -1
View File
@@ -19,7 +19,7 @@ package hdfs
import (
"strings"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/minio-go/v6/pkg/s3utils"
minio "github.com/minio/minio/cmd"
)
+36 -59
View File
@@ -29,9 +29,9 @@ import (
"syscall"
"time"
"github.com/colinmarc/hdfs/v2"
"github.com/minio/cli"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/hdfs/v3"
"github.com/minio/minio-go/v6/pkg/s3utils"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
@@ -76,18 +76,18 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for HDFS backend.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ {{.HelpName}} hdfs://namenode:8200
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}} hdfs://namenode:8200
2. Start minio gateway server for HDFS with edge caching enabled.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ {{.HelpName}} hdfs://namenode:8200
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXPIRY{{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_MAXUSE{{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}} hdfs://namenode:8200
`
minio.RegisterGatewayCommand(cli.Command{
@@ -287,24 +287,9 @@ func (n *hdfsObjects) ListBuckets(ctx context.Context) (buckets []minio.BucketIn
return buckets, nil
}
func (n *hdfsObjects) isObjectDir(bucket, object string) bool {
f, err := n.clnt.Open(minio.PathJoin(hdfsSeparator, bucket, object))
if err != nil {
return false
}
defer f.Close()
entries, err := f.Readdir(1)
if err != nil {
return false
}
return len(entries) == 0
}
func (n *hdfsObjects) listDirFactory(isLeaf minio.IsLeafFunc) minio.ListDirFunc {
func (n *hdfsObjects) listDirFactory() minio.ListDirFunc {
// listDir - lists all the entries at a given prefix and given entry in the prefix.
listDir := func(bucket, prefixDir, prefixEntry string) (entries []string, delayIsLeaf bool) {
listDir := func(bucket, prefixDir, prefixEntry string) (entries []string) {
f, err := n.clnt.Open(minio.PathJoin(hdfsSeparator, bucket, prefixDir))
if err != nil {
if os.IsNotExist(err) {
@@ -327,8 +312,7 @@ func (n *hdfsObjects) listDirFactory(isLeaf minio.IsLeafFunc) minio.ListDirFunc
}
}
fis = nil
entries, delayIsLeaf = minio.FilterListEntries(bucket, prefixDir, entries, prefixEntry, isLeaf)
return entries, delayIsLeaf
return minio.FilterMatchingPrefix(entries, prefixEntry)
}
// Return list factory instance.
@@ -337,27 +321,11 @@ func (n *hdfsObjects) listDirFactory(isLeaf minio.IsLeafFunc) minio.ListDirFunc
// ListObjects lists all blobs in HDFS bucket filtered by prefix.
func (n *hdfsObjects) ListObjects(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi minio.ListObjectsInfo, err error) {
isLeaf := func(bucket, object string) bool {
// bucket argument is unused as we don't need to StatFile
// to figure if it's a file, just need to check that the
// object string does not end with "/".
return !strings.HasSuffix(object, hdfsSeparator)
}
// Return true if the specified object is an empty directory
isLeafDir := func(bucket, object string) bool {
if !strings.HasSuffix(object, hdfsSeparator) {
return false
}
return n.isObjectDir(bucket, object)
}
listDir := n.listDirFactory(isLeaf)
getObjectInfo := func(ctx context.Context, bucket, entry string) (minio.ObjectInfo, error) {
return n.GetObjectInfo(ctx, bucket, entry, minio.ObjectOptions{})
}
return minio.ListObjects(ctx, n, bucket, prefix, marker, delimiter, maxKeys, n.listPool, isLeaf, isLeafDir, listDir, getObjectInfo, getObjectInfo)
return minio.ListObjects(ctx, n, bucket, prefix, marker, delimiter, maxKeys, n.listPool, n.listDirFactory(), getObjectInfo, getObjectInfo)
}
// Check if the given error corresponds to ENOTEMPTY for unix
@@ -432,6 +400,14 @@ func (n *hdfsObjects) DeleteObject(ctx context.Context, bucket, object string) e
return hdfsToObjectErr(ctx, n.deleteObject(minio.PathJoin(hdfsSeparator, bucket), minio.PathJoin(hdfsSeparator, bucket, object)), bucket, object)
}
func (n *hdfsObjects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = n.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
func (n *hdfsObjects) GetObjectNInfo(ctx context.Context, bucket, object string, rs *minio.HTTPRangeSpec, h http.Header, lockType minio.LockType, opts minio.ObjectOptions) (gr *minio.GetObjectReader, err error) {
objInfo, err := n.GetObjectInfo(ctx, bucket, object, opts)
if err != nil {
@@ -477,13 +453,15 @@ func (n *hdfsObjects) GetObject(ctx context.Context, bucket, key string, startOf
if err != nil {
return hdfsToObjectErr(ctx, err, bucket, key)
}
if _, err = rd.Seek(startOffset, io.SeekStart); err != nil {
return hdfsToObjectErr(ctx, err, bucket, key)
defer rd.Close()
_, err = io.Copy(writer, io.NewSectionReader(rd, startOffset, length))
if err == io.ErrClosedPipe {
// hdfs library doesn't send EOF correctly, so io.Copy attempts
// to write which returns io.ErrClosedPipe - just ignore
// this for now.
err = nil
}
if _, err = io.Copy(writer, io.LimitReader(rd, length)); err != nil {
return hdfsToObjectErr(ctx, err, bucket, key)
}
return nil
return hdfsToObjectErr(ctx, err, bucket, key)
}
// GetObjectInfo reads object info and replies back ObjectInfo.
@@ -528,17 +506,19 @@ func (n *hdfsObjects) PutObject(ctx context.Context, bucket string, object strin
return objInfo, hdfsToObjectErr(ctx, err, bucket, object)
}
defer n.deleteObject(minio.PathJoin(hdfsSeparator, minioMetaTmpBucket), tmpname)
defer w.Close()
if _, err = io.Copy(w, r); err != nil {
w.Close()
return objInfo, hdfsToObjectErr(ctx, err, bucket, object)
}
dir := path.Dir(name)
if dir != "" {
if err = n.clnt.MkdirAll(dir, os.FileMode(0755)); err != nil {
w.Close()
n.deleteObject(minio.PathJoin(hdfsSeparator, bucket), dir)
return objInfo, hdfsToObjectErr(ctx, err, bucket, object)
}
}
w.Close()
if err = n.clnt.Rename(tmpname, name); err != nil {
return objInfo, hdfsToObjectErr(ctx, err, bucket, object)
}
@@ -675,10 +655,7 @@ func (n *hdfsObjects) CompleteMultipartUpload(ctx context.Context, bucket, objec
}
// Calculate s3 compatible md5sum for complete multipart.
s3MD5, err := minio.GetCompleteMultipartMD5(ctx, parts)
if err != nil {
return objInfo, err
}
s3MD5 := minio.ComputeCompleteMultipartMD5(parts)
return minio.ObjectInfo{
Bucket: bucket,
+10 -10
View File
@@ -60,18 +60,18 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for NAS backend.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ {{.HelpName}} /shared/nasvol
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}} /shared/nasvol
2. Start minio gateway server for NAS with edge caching enabled.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ {{.HelpName}} /shared/nasvol
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXPIRY{{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_MAXUSE{{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}} /shared/nasvol
`
minio.RegisterGatewayCommand(cli.Command{
+23 -15
View File
@@ -29,8 +29,8 @@ import (
humanize "github.com/dustin/go-humanize"
"github.com/minio/cli"
miniogopolicy "github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio-go/pkg/s3utils"
miniogopolicy "github.com/minio/minio-go/v6/pkg/policy"
"github.com/minio/minio-go/v6/pkg/s3utils"
minio "github.com/minio/minio/cmd"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
@@ -79,23 +79,23 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for Aliyun OSS backend.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}}
2. Start minio gateway server for Aliyun OSS backend on custom endpoint.
$ export MINIO_ACCESS_KEY=Q3AM3UQ867SPQQA43P2F
$ export MINIO_SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
$ {{.HelpName}} https://oss.example.com
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}Q3AM3UQ867SPQQA43P2F
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
{{.Prompt}} {{.HelpName}} https://oss.example.com
3. Start minio gateway server for Aliyun OSS backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_ACCESS_KEY{{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_SECRET_KEY{{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_DRIVES{{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXCLUDE{{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_EXPIRY{{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}} MINIO_CACHE_MAXUSE{{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}}
`
minio.RegisterGatewayCommand(cli.Command{
@@ -710,6 +710,14 @@ func (l *ossObjects) DeleteObject(ctx context.Context, bucket, object string) er
return nil
}
func (l *ossObjects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = l.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
// fromOSSClientListMultipartsInfo converts oss ListMultipartUploadResult to ListMultipartsInfo
func fromOSSClientListMultipartsInfo(lmur oss.ListMultipartUploadResult) minio.ListMultipartsInfo {
uploads := make([]minio.MultipartInfo, len(lmur.Uploads))
+1 -1
View File
@@ -241,7 +241,7 @@ func getGWMetadata(ctx context.Context, bucket, prefix string, gwMeta gwMetaV1)
logger.LogIf(ctx, err)
return nil, err
}
hashReader, err := hash.NewReader(bytes.NewReader(metadataBytes), int64(len(metadataBytes)), "", "", int64(len(metadataBytes)))
hashReader, err := hash.NewReader(bytes.NewReader(metadataBytes), int64(len(metadataBytes)), "", "", int64(len(metadataBytes)), false)
if err != nil {
return nil, err
}
+3 -2
View File
@@ -26,7 +26,7 @@ import (
"strings"
"time"
"github.com/minio/minio-go/pkg/encrypt"
"github.com/minio/minio-go/v6/pkg/encrypt"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/cmd/logger"
@@ -442,7 +442,8 @@ func (l *s3EncObjects) PutObject(ctx context.Context, bucket string, object stri
// Decide if sse options needed to be passed to backend
if opts.ServerSideEncryption != nil &&
((minio.GlobalGatewaySSE.SSEC() && opts.ServerSideEncryption.Type() == encrypt.SSEC) ||
(minio.GlobalGatewaySSE.SSES3() && opts.ServerSideEncryption.Type() == encrypt.S3)) {
(minio.GlobalGatewaySSE.SSES3() && opts.ServerSideEncryption.Type() == encrypt.S3) ||
opts.ServerSideEncryption.Type() == encrypt.KMS) {
sseOpts = opts.ServerSideEncryption
}
if opts.ServerSideEncryption == nil {
+66 -38
View File
@@ -27,12 +27,12 @@ import (
"time"
"github.com/minio/cli"
miniogo "github.com/minio/minio-go"
"github.com/minio/minio-go/pkg/credentials"
miniogo "github.com/minio/minio-go/v6"
"github.com/minio/minio-go/v6/pkg/credentials"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio-go/pkg/encrypt"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/minio-go/v6/pkg/encrypt"
"github.com/minio/minio-go/v6/pkg/s3utils"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/policy"
@@ -77,38 +77,38 @@ ENVIRONMENT VARIABLES:
EXAMPLES:
1. Start minio gateway server for AWS S3 backend.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}}
2. Start minio gateway server for S3 backend on custom endpoint.
$ export MINIO_ACCESS_KEY=Q3AM3UQ867SPQQA43P2F
$ export MINIO_SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
$ {{.HelpName}} https://play.min.io:9000
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}Q3AM3UQ867SPQQA43P2F
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
{{.Prompt}} {{.HelpName}} https://play.min.io:9000
3. Start minio gateway server for AWS S3 backend logging all requests to http endpoint.
$ export MINIO_ACCESS_KEY=Q3AM3UQ867SPQQA43P2F
$ export MINIO_SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
$ export MINIO_LOGGER_HTTP_ENDPOINT="http://localhost:8000/"
$ {{.HelpName}} https://play.min.io:9000
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}Q3AM3UQ867SPQQA43P2F
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}"http://localhost:8000/"
{{.Prompt}} {{.HelpName}} https://play.min.io:9000
4. Start minio gateway server for AWS S3 backend with edge caching enabled.
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
$ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}secretkey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}"/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}"bucket1/*;*.png"
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}40
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}80
{{.Prompt}} {{.HelpName}}
4. Start minio gateway server for AWS S3 backend using AWS environment variables.
NOTE: The access and secret key in this case will authenticate with MinIO instead
of AWS and AWS envs will be used to authenticate to AWS S3.
$ export AWS_ACCESS_KEY_ID=aws_access_key
$ export AWS_SECRET_ACCESS_KEY=aws_secret_key
$ export MINIO_ACCESS_KEY=accesskey
$ export MINIO_SECRET_KEY=secretkey
$ {{.HelpName}}
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}aws_access_key
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}aws_secret_key
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}accesskey
{{.Prompt}} {{.EnvVarSetCommand}}(.*){{.AssignmentOperator}}secretkey
{{.Prompt}} {{.HelpName}}
`
minio.RegisterGatewayCommand(cli.Command{
@@ -169,14 +169,6 @@ func randString(n int, src rand.Source, prefix string) string {
return prefix + string(b[0:30-len(prefix)])
}
func isAmazonS3Endpoint(urlStr string) bool {
u, err := url.Parse(urlStr)
if err != nil {
panic(err)
}
return s3utils.IsAmazonEndpoint(*u)
}
// Chains all credential types, in the following order:
// - AWS env vars (i.e. AWS_ACCESS_KEY_ID)
// - AWS creds file (i.e. AWS_SHARED_CREDENTIALS_FILE or ~/.aws/credentials)
@@ -210,6 +202,11 @@ func newS3(urlStr string) (*miniogo.Core, error) {
urlStr = "https://s3.amazonaws.com"
}
u, err := url.Parse(urlStr)
if err != nil {
return nil, err
}
// Override default params if the host is provided
endpoint, secure, err := minio.ParseGatewayEndpoint(urlStr)
if err != nil {
@@ -217,15 +214,23 @@ func newS3(urlStr string) (*miniogo.Core, error) {
}
var creds *credentials.Credentials
if isAmazonS3Endpoint(urlStr) {
if s3utils.IsAmazonEndpoint(*u) {
// If we see an Amazon S3 endpoint, then we use more ways to fetch backend credentials.
// Specifically IAM style rotating credentials are only supported with AWS S3 endpoint.
creds = credentials.NewChainCredentials(defaultAWSCredProviders)
} else {
creds = credentials.NewChainCredentials(defaultProviders)
}
clnt, err := miniogo.NewWithCredentials(endpoint, creds, secure, "")
options := miniogo.Options{
Creds: creds,
Secure: secure,
Region: s3utils.GetRegionFromURL(*u),
BucketLookup: miniogo.BucketLookupAuto,
}
clnt, err := miniogo.NewWithOptions(endpoint, &options)
if err != nil {
return nil, err
}
@@ -234,9 +239,12 @@ func newS3(urlStr string) (*miniogo.Core, error) {
clnt.SetCustomTransport(minio.NewCustomHTTPTransport())
probeBucketName := randString(60, rand.NewSource(time.Now().UnixNano()), "probe-bucket-sign-")
// Check if the provided keys are valid.
if _, err = clnt.BucketExists(probeBucketName); err != nil {
return nil, err
if miniogo.ToErrorResponse(err).Code != "AccessDenied" {
return nil, err
}
}
return &miniogo.Core{Client: clnt}, nil
@@ -313,7 +321,19 @@ func (l *s3Objects) MakeBucketWithLocation(ctx context.Context, bucket, location
func (l *s3Objects) GetBucketInfo(ctx context.Context, bucket string) (bi minio.BucketInfo, e error) {
buckets, err := l.Client.ListBuckets()
if err != nil {
return bi, minio.ErrorRespToObjectError(err, bucket)
// Listbuckets may be disallowed, proceed to check if
// bucket indeed exists, if yes return success.
var ok bool
if ok, err = l.Client.BucketExists(bucket); err != nil {
return bi, minio.ErrorRespToObjectError(err, bucket)
}
if !ok {
return bi, minio.BucketNotFound{Bucket: bucket}
}
return minio.BucketInfo{
Name: bi.Name,
Created: time.Now().UTC(),
}, nil
}
for _, bi := range buckets {
@@ -500,6 +520,14 @@ func (l *s3Objects) DeleteObject(ctx context.Context, bucket string, object stri
return nil
}
func (l *s3Objects) DeleteObjects(ctx context.Context, bucket string, objects []string) ([]error, error) {
errs := make([]error, len(objects))
for idx, object := range objects {
errs[idx] = l.DeleteObject(ctx, bucket, object)
}
return errs, nil
}
// ListMultipartUploads lists all multipart uploads.
func (l *s3Objects) ListMultipartUploads(ctx context.Context, bucket string, prefix string, keyMarker string, uploadIDMarker string, delimiter string, maxUploads int) (lmi minio.ListMultipartsInfo, e error) {
result, err := l.Client.ListMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
+1 -1
View File
@@ -20,7 +20,7 @@ import (
"fmt"
"testing"
miniogo "github.com/minio/minio-go"
miniogo "github.com/minio/minio-go/v6"
"github.com/minio/minio/pkg/hash"
minio "github.com/minio/minio/cmd"
+27 -57
View File
@@ -24,16 +24,15 @@ import (
"strings"
"time"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio-go/v6/pkg/set"
humanize "github.com/dustin/go-humanize"
"github.com/minio/minio/cmd/crypto"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/dns"
"github.com/minio/minio/pkg/handlers"
"github.com/minio/minio/pkg/sys"
"github.com/rs/cors"
"golang.org/x/time/rate"
)
// HandlerFunc - useful to chain different middleware http.Handler
@@ -263,10 +262,10 @@ func (h cacheControlHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if hasSuffix(r.URL.Path, ".js") || r.URL.Path == minioReservedBucketPath+"/favicon.ico" {
// For assets set cache expiry of one year. For each release, the name
// of the asset name will change and hence it can not be served from cache.
w.Header().Set("Cache-Control", "max-age=31536000")
w.Header().Set(xhttp.CacheControl, "max-age=31536000")
} else {
// For non asset requests we serve index.html which will never be cached.
w.Header().Set("Cache-Control", "no-store")
w.Header().Set(xhttp.CacheControl, "no-store")
}
}
}
@@ -296,7 +295,7 @@ func (h minioReservedBucketHandler) ServeHTTP(w http.ResponseWriter, r *http.Req
default:
// For all other requests reject access to reserved
// buckets
bucketName, _ := urlPath2BucketObjectName(r.URL.Path)
bucketName, _ := request2BucketObjectName(r)
if isMinioReservedBucket(bucketName) || isMinioMetaBucket(bucketName) {
writeErrorResponse(context.Background(), w, errorCodes.ToAPIErr(ErrAllAccessDisabled), r.URL, guessIsBrowserReq(r))
return
@@ -381,6 +380,20 @@ type resourceHandler struct {
// setCorsHandler handler for CORS (Cross Origin Resource Sharing)
func setCorsHandler(h http.Handler) http.Handler {
commonS3Headers := []string{
xhttp.Date,
xhttp.ETag,
xhttp.ServerInfo,
xhttp.Connection,
xhttp.AcceptRanges,
xhttp.ContentRange,
xhttp.ContentEncoding,
xhttp.ContentLength,
xhttp.ContentType,
"X-Amz*",
"x-amz*",
"*",
}
c := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
@@ -393,8 +406,8 @@ func setCorsHandler(h http.Handler) http.Handler {
http.MethodOptions,
http.MethodPatch,
},
AllowedHeaders: []string{"*"},
ExposedHeaders: []string{"*"},
AllowedHeaders: commonS3Headers,
ExposedHeaders: commonS3Headers,
AllowCredentials: true,
})
@@ -482,7 +495,7 @@ var notimplementedObjectResourceNames = map[string]bool{
// Resource handler ServeHTTP() wrapper
func (h resourceHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
bucketName, objectName := urlPath2BucketObjectName(r.URL.Path)
bucketName, objectName := request2BucketObjectName(r)
// If bucketName is present and not objectName check for bucket level resource queries.
if bucketName != "" && objectName == "" {
@@ -684,7 +697,8 @@ func (f bucketForwardingHandler) ServeHTTP(w http.ResponseWriter, r *http.Reques
return
}
bucket, object := urlPath2BucketObjectName(r.URL.Path)
bucket, object := request2BucketObjectName(r)
// ListBucket requests should be handled at current endpoint as
// all buckets data can be fetched from here.
if r.Method == http.MethodGet && bucket == "" && object == "" {
@@ -737,47 +751,7 @@ func setBucketForwardingHandler(h http.Handler) http.Handler {
return bucketForwardingHandler{fwd, h}
}
// setRateLimitHandler middleware limits the throughput to h using a
// rate.Limiter token bucket configured with maxOpenFileLimit and
// burst set to 1. The request will idle for up to 1*time.Second.
// If the limiter detects the deadline will be exceeded, the request is
// canceled immediately.
func setRateLimitHandler(h http.Handler) http.Handler {
_, maxLimit, err := sys.GetMaxOpenFileLimit()
logger.FatalIf(err, "Unable to get maximum open file limit")
// Burst value is set to 1 to allow only maxOpenFileLimit
// requests to happen at once.
l := rate.NewLimiter(rate.Limit(maxLimit), 1)
return rateLimit{l, h}
}
type rateLimit struct {
*rate.Limiter
handler http.Handler
}
func (l rateLimit) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// create a new context from the request with the wait timeout
ctx, cancel := context.WithTimeout(r.Context(), 1*time.Second)
defer cancel() // always cancel the context!
// Wait errors out if the request cannot be processed within
// the deadline. time/rate tries to reserve a slot if possible
// with in the given duration if it's not possible then Wait(ctx)
// returns an error and we cancel the request with ErrSlowDown
// error message to the client. This context wait also ensures
// requests doomed to fail are terminated early, preventing a
// potential pileup on the server.
if err := l.Wait(ctx); err != nil {
// Send an S3 compatible error, SlowDown.
writeErrorResponse(context.Background(), w, errorCodes.ToAPIErr(ErrSlowDown), r.URL, guessIsBrowserReq(r))
return
}
l.handler.ServeHTTP(w, r)
}
// customHeaderHandler sets x-amz-request-id, x-minio-deployment-id header.
// customHeaderHandler sets x-amz-request-id header.
// Previously, this value was set right before a response was sent to
// the client. So, logger and Error response XML were not using this
// value. This is set here so that this header can be logged as
@@ -791,12 +765,8 @@ func addCustomHeaders(h http.Handler) http.Handler {
}
func (s customHeaderHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Set custom headers such as x-amz-request-id and x-minio-deployment-id
// for each request.
w.Header().Set(responseRequestIDKey, mustGetRequestID(UTCNow()))
if globalDeploymentID != "" {
w.Header().Set(responseDeploymentIDKey, globalDeploymentID)
}
// Set custom headers such as x-amz-request-id for each request.
w.Header().Set(xhttp.AmzRequestID, mustGetRequestID(UTCNow()))
s.handler.ServeHTTP(logger.NewResponseWriter(w), r)
}
+17 -15
View File
@@ -23,7 +23,7 @@ import (
"time"
isatty "github.com/mattn/go-isatty"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio-go/v6/pkg/set"
etcd "github.com/coreos/etcd/clientv3"
humanize "github.com/dustin/go-humanize"
@@ -35,6 +35,7 @@ import (
"github.com/minio/minio/pkg/dns"
iampolicy "github.com/minio/minio/pkg/iam/policy"
"github.com/minio/minio/pkg/iam/validator"
"github.com/minio/minio/pkg/pubsub"
)
// minio configuration related constants.
@@ -82,8 +83,6 @@ const (
// GlobalMultipartCleanupInterval - Cleanup interval when the stale multipart cleanup is initiated.
GlobalMultipartCleanupInterval = time.Hour * 24 // 24 hrs.
// Refresh interval to update in-memory bucket policy cache.
globalRefreshBucketPolicyInterval = 5 * time.Minute
// Refresh interval to update in-memory iam config cache.
globalRefreshIAMInterval = 5 * time.Minute
@@ -92,9 +91,10 @@ const (
)
var globalCLIContext = struct {
JSON, Quiet bool
Anonymous bool
Addr string
JSON, Quiet bool
Anonymous bool
Addr string
StrictS3Compat bool
}{}
var (
@@ -110,6 +110,9 @@ var (
// Indicates if the running minio server is an erasure-code backend.
globalIsXL = false
// Indicates if the running minio is in gateway mode.
globalIsGateway = false
// This flag is set to 'true' by default
globalIsBrowserEnabled = true
@@ -157,8 +160,9 @@ var (
globalHTTPServerErrorCh = make(chan error)
globalOSSignalCh = make(chan os.Signal, 1)
// File to log HTTP request/response headers and body.
globalHTTPTraceFile *os.File
// global Trace system to send HTTP request/response logs to
// registered listeners
globalHTTPTrace = pubsub.New()
globalEndpoints EndpointList
@@ -208,13 +212,6 @@ var (
// Max allowed disk cache percentage
globalCacheMaxUse = 80
// RPC V1 - Initial version
// RPC V2 - format.json XL version changed to 2
// RPC V3 - format.json XL version changed to 3
// RPC V4 - ReadFile() arguments signature changed
// Current RPC version
globalRPCAPIVersion = RPCVersion{4, 0, 0}
// Allocated etcd endpoint for config and bucket DNS.
globalEtcdClient *etcd.Client
@@ -265,6 +262,11 @@ var (
// GlobalGatewaySSE sse options
GlobalGatewaySSE gatewaySSE
// The always present healing routine ready to heal objects
globalBackgroundHealing *healRoutine
globalAllHealState *allHealState
globalSweepHealState *allHealState
// Add new variable global values here.
)
+17 -9
View File
@@ -27,10 +27,10 @@ import (
"net/url"
"strings"
xhttp "github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/handlers"
httptracer "github.com/minio/minio/pkg/handlers"
)
// Parses location constraint from the incoming reader.
@@ -220,8 +220,8 @@ func extractReqParams(r *http.Request) map[string]string {
func extractRespElements(w http.ResponseWriter) map[string]string {
return map[string]string{
"requestId": w.Header().Get(responseRequestIDKey),
"content-length": w.Header().Get("Content-Length"),
"requestId": w.Header().Get(xhttp.AmzRequestID),
"content-length": w.Header().Get(xhttp.ContentLength),
// Add more fields here.
}
}
@@ -326,18 +326,26 @@ func extractPostPolicyFormValues(ctx context.Context, form *multipart.Form) (fil
// Log headers and body.
func httpTraceAll(f http.HandlerFunc) http.HandlerFunc {
if globalHTTPTraceFile == nil {
return f
return func(w http.ResponseWriter, r *http.Request) {
if !globalHTTPTrace.HasSubscribers() {
f.ServeHTTP(w, r)
return
}
trace := Trace(f, true, w, r)
globalHTTPTrace.Publish(trace)
}
return httptracer.TraceReqHandlerFunc(f, globalHTTPTraceFile, true)
}
// Log only the headers.
func httpTraceHdrs(f http.HandlerFunc) http.HandlerFunc {
if globalHTTPTraceFile == nil {
return f
return func(w http.ResponseWriter, r *http.Request) {
if !globalHTTPTrace.HasSubscribers() {
f.ServeHTTP(w, r)
return
}
trace := Trace(f, false, w, r)
globalHTTPTrace.Publish(trace)
}
return httptracer.TraceReqHandlerFunc(f, globalHTTPTraceFile, false)
}
// Returns "/bucketName/objectName" for path-style or virtual-host-style requests.

Some files were not shown because too many files have changed in this diff Show More