Compare commits

...

186 Commits

Author SHA1 Message Date
Minio Trusted 1b472dae78 Bump to new release. 2017-04-28 17:58:49 -07:00
Krishna Srinivas eb50175ad9 gateway: reject bad path segments in URL (#4202) 2017-04-28 17:26:13 -07:00
Krishna Srinivas e85349381e gateway: Fix help message for gateway (#4201) 2017-04-28 17:26:00 -07:00
Krishna Srinivas 06bc68a4b3 gateway: Fix help message for custom Azure Blob Storage endpoint. (#4113) 2017-04-28 17:23:41 -07:00
Krishna Srinivas fb506c7fca gateway: Support for custom endpoint. (#4086) 2017-04-28 17:23:23 -07:00
Harshavardhana 8a7cffe7b8 docs: macOS brew now refers to Minio fork (#4059) 2017-04-25 11:01:51 -07:00
Minio Trusted 5c85ce1afd Bump to new release. 2017-04-24 18:34:34 -07:00
Harshavardhana 710db6bdad build: Reduce binary size by using -s -w (#4027)
Refer #3939
2017-04-24 18:23:42 -07:00
Harshavardhana 058ea84605 server: Validate path for bad components in a handler. (#4170) 2017-04-24 18:22:40 -07:00
Minio Trusted 6e7d33df20 Bump to new release 2017-03-16 14:52:42 -07:00
Minio Trusted 5311eb22fd Fix dockerfile 2017-03-16 14:49:30 -07:00
Krishna Srinivas cea4cfa3a8 Implement S3 Gateway to third party cloud storage providers. (#3756)
Currently supported backend is Azure Blob Storage.

```
export MINIO_ACCESS_KEY=azureaccountname
export MINIO_SECRET_KEY=azureaccountkey
minio gateway azure
```
2017-03-16 12:21:58 -07:00
Anis Elleuch 8426cf9aec config: Accept more address format + unit test (#3915)
checkURL() is a generic function to check if a passed address
is valid. This commit adds support for addresses like `m1`
and `172.16.3.1` which is needed in MySQL and NATS. This commit
also adds tests.
2017-03-16 11:44:01 -07:00
Harshavardhana f3334159a4 config: Relax browser and region to be empty. (#3912)
- If browser field is missing or empty
  then default to "on".

- If region field is empty or missing then
  default to "us-east-1" (S3 spec behavior)
2017-03-16 11:06:17 -07:00
Harshavardhana 3edff1501e Vendor upstream redis library instead of our fork. (#3913)
We forked the upstream to address a build issue on
go 1.6 - that is long done and we don't need
to manage our forks anymore.
2017-03-16 08:22:47 -07:00
Bala FA 21d73a3eef Simplify credential usage. (#3893) 2017-03-16 00:16:06 -07:00
Krishnan Parthasarathi 051f9bb5c6 Implement list uploads heal admin API (#3885) 2017-03-16 00:15:06 -07:00
Harshavardhana 6509589adb Use canonicalETag helper wherever needed. (#3910) 2017-03-15 20:48:49 -07:00
Anis Elleuch ae4361cc45 config: Check for duplicated entries in all scopes (#3872)
Validate Minio config by checking if there is double json key
in any scope level. The returned error contains the json path
to the duplicated key.
2017-03-15 16:30:34 -07:00
Krishna Srinivas cad0d0eb7a browser: Update ui-assets.go (#3902)
fixes #3898
2017-03-15 11:32:51 -07:00
Nitish Tiwari ba0c11757e Added that no special config changes reqd (#3906)
Added a line saying no special config changes are required for Shared mode. Also the previous `Why Shared Backend` and current `Use cases` are already merged. This fixes the comment: https://github.com/minio/minio/pull/3888#discussion_r105573494
2017-03-15 08:22:03 -07:00
Krishna Srinivas 96050c1e21 browser: Do not show "Loading..." if there are no buckets. (#3904) 2017-03-14 19:07:23 -07:00
Rushan a27d1b3d86 Browser: Use babel-polyfill to support new ES6 built-ins in older browsers (#3900) 2017-03-14 15:02:20 -07:00
Anis Elleuch a5e60706a2 xl,fs: Return 404 if object ends with a separator (#3897)
HEAD Object for FS and XL was returning invalid object name when
an object name has a trailing slash separator, this PR changes the
behavior and will always return 404 object not found, this guarantees
a better compatibility with S3 spec.
2017-03-13 22:20:46 -07:00
Harshavardhana 5f7565762e api: postPolicy cleanup. Simplify the code and re-use. (#3890)
This change is cleanup of the postPolicyHandler code
primarily to address the flow and also converting
certain critical parts into self contained functions.
2017-03-13 14:41:13 -07:00
Nitish Tiwari 3314501f19 Simplify shared mode document (#3888) 2017-03-12 16:17:03 -07:00
Harshavardhana 9d53a646a1 Simplify the title for orchestration and some words. (#3887) 2017-03-11 02:17:03 -08:00
Nitish Tiwari 5e0032e165 Update links (#3886) 2017-03-11 00:11:07 -08:00
Harshavardhana 305952d734 browser: Update ui-assets with new changes. 2017-03-10 15:13:26 -08:00
Rushan e77885d671 Browser: Add Object.assign polyfill to support older browsers (#3884) 2017-03-10 14:30:23 -08:00
Nitish Tiwari 2410eb281e Update kafkacat command with consumer flag (#3882) 2017-03-10 09:01:59 -08:00
Harshavardhana e54025805f browser: update ui-assets with new changes. 2017-03-09 15:26:25 -08:00
Anis Elleuch d602495600 madmin: Do not require SSL to set credentials (#3879)
We need to relax this requirement and let the client decides
if it can allow to set credentials API over plain connection.
2017-03-09 14:08:33 -08:00
Nitish Tiwari 03937e7554 Added documentation for orchestration platforms (#3684) 2017-03-09 14:06:51 -08:00
Harshavardhana e3b627a192 docker: Add ARM64 image build support (#3876) 2017-03-09 13:57:27 -08:00
Rushan ccc3349f0c Browser: Show complete bucket name by removing the ellipsis cut off (#3877) 2017-03-09 10:42:38 -08:00
Harshavardhana 85cbd875fc cleanup: All conditionals simplified under pkg. (#3875)
Address all the changes reported/recommended by
`gosimple` tool.
2017-03-09 10:13:30 -08:00
Harshavardhana 43317530d5 Fix odd shadowing bug in XL init. (#3874)
Fixes #3873
2017-03-08 20:42:45 -08:00
Bala FA 8a9852220d Make unit testable cert parsing functions. (#3863) 2017-03-08 19:20:01 -08:00
Harshavardhana 47ac410ab0 Code cleanup - simplify server side code. (#3870)
Fix all the issues reported by `gosimple` tool.
2017-03-08 10:00:47 -08:00
Harshavardhana 433225ab0d docker: ca-certificates should not be removed. (#3868) 2017-03-07 16:30:19 -08:00
Harshavardhana 3e655a2c85 docker: Add ARM docker container dockerfile. (#3574) 2017-03-07 15:15:05 -08:00
Anis Elleuch a2eae54d11 xl: Respect min. space by checking PrepareFile err (#3867)
It was possible to upload a big file which overcomes the minimal
disk space limit in XL, PrepareFile was actually checking for disk
space but we weren't checking its returned error. This patch fixes
this behavior.
2017-03-07 14:48:56 -08:00
Anis Elleuch 79e0b9e69a Relax minio server start when disk threshold is reached and adds space check in FS (#3865)
* fs: Rename tempObjPath variable in fsCreateFile()
* fs/posix: Factor checkDiskFree() function
* fs: Add disk free check in fsCreateFile()
* posix: Move free disk check to createFile()
* xl: Relax free disk check in POSIX initialization
* fs: checkDiskFree checks for space to store data
2017-03-07 12:25:40 -08:00
Krishna Srinivas 29ff9674a0 browser: Humanize expiry time for Share-Object. (#3861) 2017-03-06 20:13:52 -08:00
Bala FA bff4d29415 Remove commands and commandsTree global variables. (#3855) 2017-03-06 19:35:26 -08:00
Krishna Srinivas 436db49bf3 Share object expiry value modification modal was not working (#3860) 2017-03-06 18:50:25 -08:00
Rushan 966818955e Browser: Implement multiple object delete (#3859) 2017-03-06 15:43:43 -08:00
Harshavardhana e49efcb9d9 xl: quickHeal heal bucket only when needed. (#3854)
This improves the startup time significantly
for clusters which have lot of buckets.

Also fixes a bug where `.minio.sys` is created
on disks which do not have `format.json`
2017-03-06 02:00:15 -08:00
Harshavardhana 6f931d29c4 rpm: Add RPM spec for minio build. (#3853)
Currently the package is built and hosted at

https://copr.fedorainfracloud.org/coprs/minio/minio/

To enable minio repo one has to download.

Fedora - 25
https://copr.fedorainfracloud.org/coprs/minio/minio/repo/fedora-25/minio-minio-fedora-25.repo

Fedora - 26
https://copr.fedorainfracloud.org/coprs/minio/minio/repo/fedora-26/minio-minio-fedora-26.repo

Enables for both i386 and x86_64.

Fixes #3576
2017-03-05 13:09:31 -08:00
Krishnan Parthasarathi e3fd4c0dd6 XL: Make listOnlineDisks and outDatedDisks consistent w/ each other. (#3808) 2017-03-04 14:53:28 -08:00
Harshavardhana b05c1c11d4 browser: Update UI assets with new changes. 2017-03-03 18:05:02 -08:00
Krishna Srinivas 0bae3330e8 browser: Send correct arguments for RemoveObjects web handler. (#3848)
Fixes #3839
2017-03-03 17:07:17 -08:00
Harshavardhana 05e53f1b34 api: CopyObjectPart was copying wrong offsets due to shadowing. (#3838)
startOffset was re-assigned to '0' so it would end up
copying wrong content ignoring the requested startOffset.

This also fixes the corruption issue we observed while
using docker registry.

Fixes https://github.com/docker/distribution/issues/2205

Also fixes #3842 - incorrect routing.
2017-03-03 16:32:04 -08:00
Anis Elleuch 0c8c463a63 tests: Fix web handlers testing with faulty disks (#3845) 2017-03-03 15:46:28 -08:00
Aditya Manthramurthy 6df7bc42b8 Fix check for bucket name: (#3832)
* Do not allow bucket names with adjacent hypen and periods.
* Improve performance by eliminating the usage of regular expressions.
2017-03-03 10:23:41 -08:00
Anis Elleuch 6c00a57a7c quick: Add yaml format support (#3833)
quick Save() and Load() infers config file's format from
file name extension.
2017-03-03 10:22:09 -08:00
Harshavardhana bc52d911ef api: Increase the maximum object size limit from 5GiB to 16GiB. (#3834)
The globalMaxObjectSize limit is instilled in S3 spec perhaps
due to certain limitations on S3 infrastructure. For minio we
don't have such limitations and we can stream a larger file
instead.

So we are going to bump this limit to 16GiB.

Fixes #3825
2017-03-03 10:14:17 -08:00
Anis Elleuch 28c53a3555 obj: Make checkBucketExist() returns all errors (#3843)
This function was returning BucketNotFound for all errors
which at least hides the fact that disks could be corrupted.
This commit fixes the behavior by returning all errors that,
are, by the way, Object API errors.
2017-03-03 10:12:43 -08:00
Harshavardhana e5d4e7aa9d web: Validate if bucket names are reserved (#3841)
Both '.minio.sys' and 'minio' should be never allowed
to be created from web-ui and then fail to list it
by filtering them out.

Fixes #3840
2017-03-03 03:01:42 -08:00
Anis Elleuch cddc684559 admin: Set Config returns errSet and errMsg (#3822)
There is no way to see if a node encountered an error
when trying to set a new config set, this commit adds
a bool errSet field.
2017-03-03 02:53:48 -08:00
Zejun Li 32d0d3d4ac Enhanced newObjectLayerFn (#3837) 2017-03-03 01:07:45 -08:00
Bala FA 98d17d2a97 Remove globalQuiet and globalConfigDir global variables (#3830) 2017-03-02 14:21:30 -08:00
Bala FA 208dd15245 Remove globalMaxCacheSize and globalCacheExpiry variables (#3826)
This patch fixes below

* Remove global variables globalMaxCacheSize and globalCacheExpiry.
* Make global variables into constant in objcache package.
2017-03-02 10:34:37 -08:00
Anis Elleuch a179fc9658 quick: Simplify Load() and CheckVersion() (#3831) 2017-03-02 10:29:06 -08:00
Zejun Li d1afd16955 Using RWMutex to guard closing and listeners (#3829) 2017-03-02 10:00:22 -08:00
Bala FA 2348ae7a19 Make default values as constants (#3828) 2017-03-02 04:58:39 -08:00
Bala FA 480ea826dc Move rlimit functions into sys package. (#3824)
This patch addresses below

* go build works for bsd family
* probe total RAM size for bsd family
* make unit testable functions
2017-03-01 21:51:57 -08:00
Aditya Manthramurthy 09e9fd745c Close client connection after checking for release update (#3820) 2017-03-01 09:18:55 -08:00
Anis Elleuch 77c1998a38 config: Fix creating new config with wrong version (#3821)
Simplify a little config code to avoid making mistake
next time.
2017-03-01 09:17:04 -08:00
Krishna Srinivas 91cf54f895 web-handlers: Support removal of multiple objects at once. (#3810) 2017-02-28 19:07:28 -08:00
Karthic Rao 2b0ed21f08 tests: Fix test server init - cleanup (#3806) 2017-02-28 18:05:52 -08:00
Harshavardhana 472fa4a6ca api: Multi object delete should be protected. (#3814)
Add missing protection from deleting multiple objects
in parallel. Currently we are deleting objects without
proper locking through this API.

This can cause significant amount of races.
2017-02-28 18:00:24 -08:00
Bala FA 097cec676a fix: Set globalMaxCacheSize to allowable value. (#3816)
If memory resource limit and total RAM are more than 8GiB, either 50%
of memory resource limit or total RAM is set to globalMaxCacheSize.
2017-02-28 16:51:52 -08:00
Rushan fcad4a44fd Browser: Remove duplicate object entries while sorting (#3813) 2017-02-28 13:11:34 -08:00
Anis Elleuch 9b3c014bab config: Add browser parameter (#3807)
browser new parameter receives "on" or "off" parameter which is similar
to MINIO_BROWSER
2017-02-27 14:59:53 -08:00
Krishnan Parthasarathi c9619673fb Implement SetConfig admin API handler. (#3792) 2017-02-27 11:40:27 -08:00
Anis Elleuch dce0345f8f Set disk to nil after write which needs quorum (#3795)
Ignore a disk which wasn't able to successfully perform an action to
avoid eventual perturbations when the disk comes back in the middle
of write change.
2017-02-26 11:58:32 -08:00
Anis Elleuch 461b2bbd37 admin: Move SetCredentials from Service to Generic (#3805)
Setting credentials doesn't belong to service management API
anymore.
2017-02-25 11:06:08 -08:00
Bala FA 69777b654e event: use common initialization logic (#3798)
Previously creating and adding targets for each notification type was
repeated.  This patch fixes it.
2017-02-24 18:27:52 -08:00
Harshavardhana 70d2cb5f4d rpc: Remove time check for each RPC calls. (#3804)
This removal comes to avoid some redundant requirements
which are adding more problems on a production setup.

Here are the list of checks for time as they happen

 - Fresh connect (during server startup) - CORRECT
 - A reconnect after network disconnect - CORRECT
 - For each RPC call - INCORRECT.

Verifying time for each RPC aggravates a situation
where a RPC call is rejected in a sequence of events
due to enough load on a production setup. 3 second
might not be enough time window for the call to be
initiated and received by the server.
2017-02-24 18:26:56 -08:00
Harshavardhana cff45db1b9 cli: Use ADDRESS:PORT to clarify --address behavior (#3803)
Currently we document as IP:PORT which doesn't provide
if someone can use HOSTNAME:PORT. This is a change
to clarify this by calling it as ADDRESS:PORT which
encompasses both a HOSTNAME and an IP.

Fixes #3799
2017-02-24 14:19:20 -08:00
Harshavardhana bcc5b6e1ef xl: Rename getOrderedDisks as shuffleDisks appropriately. (#3796)
This PR is for readability cleanup

- getOrderedDisks as shuffleDisks
- getOrderedPartsMetadata as shufflePartsMetadata

Distribution is now a second argument instead being the
primary input argument for brevity.

Also change the usage of type casted int64(0), instead
rely on direct type reference as `var variable int64` everywhere.
2017-02-24 09:20:40 -08:00
Harshavardhana 25b5a0534f browser: Update ui-assets and fix the copyright header. (#3790) 2017-02-22 17:27:26 -08:00
Rushan 52d6678bf0 Browser: Implement multi select user interface for object listings (#3730) 2017-02-22 14:51:38 -08:00
Nitish Tiwari d8950ba7c5 Added server times note and fix Notes rendering for Doctor. (#3787) 2017-02-22 02:12:03 -08:00
Harshavardhana cc28765025 xl/multipart: Make sure to delete temp renamed object. (#3785)
Existing objects before overwrites are renamed to
temp location in completeMultipart. We make sure
that we delete it even if subsequenty calls fail.

Additionally move verifying of parent dir is a
file earlier to fail the entire operation.

Ref #3784
2017-02-21 19:43:44 -08:00
Harshavardhana fe86319c56 ci: For windows builds stick to go1.7.5 (#3786) 2017-02-21 17:24:11 -08:00
Harshavardhana 99a12613a3 update: For source builds look for absolute path. (#3780)
os.Args[0] doesn't point to absolute path we need
use exec.LookPath to find the absolute path before
sending os.Stat().
2017-02-21 01:32:05 -08:00
Nitish Tiwari 097dd7418a Remove unused erasure diagram (#3783) 2017-02-20 21:01:08 -08:00
Nitish Tiwari a7d3ea8c15 Update erasure code image (#3782) 2017-02-20 20:12:21 -08:00
Krishnan Parthasarathi 2745bf2f1f Implement ServerConfig admin REST API (#3741)
Returns a valid config.json of the setup. In case of distributed
setup, it checks if quorum or more number of nodes have the same
config.json.
2017-02-20 12:58:50 -08:00
Anis Elleuch 70d825c608 doc: Small rewrite of bucket events notif intro (#3775) 2017-02-20 12:07:27 -08:00
Harshavardhana 6b68c0170f For streaming signature do not save content-encoding in PutObject() (#3776)
Content-Encoding is set to "aws-chunked" which is an S3 specific
API value which is no meaning for an object. This is how S3
behaves as well for a streaming signature uploaded object.
2017-02-20 12:07:03 -08:00
Aditya Manthramurthy 0a905e1a8a Fix rabbitmq reconnect problem (#3778) 2017-02-20 12:05:21 -08:00
Harshavardhana 9eb8e375c5 cli: Make sure to add --help flag for subcommands. (#3773)
--help is now back and prints properly with command
help template.
2017-02-19 20:46:06 -08:00
Harshavardhana 7ea1de8245 copyObject: Be case sensitive for windows only server. (#3766)
For case sensitive platforms we should honor case.

Fixes #3765

```
1) python s3cmd -c s3cfg_localminio put logo.png s3://testbucket/xyz/etc2/logo.PNG

2) python s3cmd -c s3cfg_localminio ls s3://testbucket/xyz/etc2/
2017-02-18 10:58     22059   s3://testbucket/xyz/etc2/logo.PNG

3) python s3cmd -c s3cfg_localminio cp s3://testbucket/xyz/etc2/logo.PNG s3://testbucket/xyz/etc2/logo.png
remote copy: 's3://testbucket/xyz/etc2/logo.PNG' -> 's3://testbucket/xyz/etc2/logo.png'

4) python s3cmd -c s3cfg_localminio ls s3://testbucket/xyz/etc2/
2017-02-18 10:58     22059   s3://testbucket/xyz/etc2/logo.PNG
2017-02-18 11:10     22059   s3://testbucket/xyz/etc2/logo.png
```
2017-02-18 13:41:59 -08:00
Anis Elleuch 54a18592e9 flags: Fix --version output (#3772) 2017-02-18 13:41:33 -08:00
Anis Elleuch 7e84c7427d server-mux: Rewrite graceful shutdown mechanism (#3771)
Old code uses waitgroup Add() and Wait() in different threads,
which eventually can lead to a race.
2017-02-18 13:28:54 -08:00
Bala FA d12f3e06b1 config-old: Use interface to avoid code repetition. (#3769) 2017-02-18 10:45:37 -08:00
Harshavardhana 0137ff498a auth/rpc: Token can be concurrently edited protect it. (#3764)
Make sure we protect when we access `authToken` in authClient.

Fixes #3761
2017-02-18 03:15:42 -08:00
Harshavardhana 34d9a6b46a Make sure client initializes to proper lock RPC path. (#3763)
Fixes a regression introduced in previous commit.
2017-02-18 02:52:11 -08:00
Harshavardhana 50b4e54a75 fs: Do not return reservedBucket names in ListBuckets() (#3754)
Make sure to skip reserved bucket names in `ListBuckets()`
current code didn't skip this properly and also generalize
this behavior for both XL and FS.
2017-02-16 14:52:14 -08:00
Harshavardhana 8816b08aae Fix the systemd config path to the new URL 2017-02-15 21:28:06 -08:00
Harshavardhana 271e3ecde5 Fix tests from cli changes 2017-02-15 18:05:55 -08:00
Minio Trusted 83abb310b4 Bump to new release 2017-02-15 17:51:19 -08:00
Minio Trusted 3d98311d9f Merge branch 'master' into release 2017-02-15 17:47:12 -08:00
Harshavardhana 611bd68739 cli: Add new features from CLI.
use `.HelpName` for template to pass down
proper command names. Also treat flags
to be optional in all of them.
2017-02-15 17:45:11 -08:00
Harshavardhana 25eeb88a8f cli: Bring upstream changes to minio server. 2017-02-15 17:30:31 -08:00
Minio Trusted d44e38b462 Bump to new release 2017-02-15 15:09:15 -08:00
Minio Trusted 8948c972ee Merge branch 'master' into release 2017-02-15 15:07:51 -08:00
Krishna Srinivas 3e770defae browser: Update ui assets with new changes (#3751) 2017-02-15 14:44:34 -08:00
Krishna Srinivas ae4656e699 browser: Remove currentPath prefix in object names from list result. (#3750) 2017-02-15 13:07:47 -08:00
Harshavardhana f6a9e690bf browser: Update ui assets with new changes. 2017-02-15 11:29:23 -08:00
Anis Elleuch 7f86a21317 admin: Add ServerInfo API() (#3743) 2017-02-15 10:45:45 -08:00
Harshavardhana fb39c7c26b sRPC/client: Properly trim storageRPCPath for actual disk path. (#3749)
Never print internal RPC endpoint paths.
2017-02-15 03:47:47 -08:00
Harshavardhana 13c3b9cbcb main/cli: Don't print hidden flags/commands in help template. (#3748)
Always use .VisibleFlags and .VisibleCommands to not print
Hidden flags as expected from help template.
2017-02-15 02:25:38 -08:00
Bala FA 602dac8773 mainUpdate: refactor to handle quiet flag properly (#3744) 2017-02-15 00:31:00 -08:00
Dee Koder c6e76160ad Update README.md 2017-02-14 14:20:29 -08:00
Bala FA a53b909fcd mainVersion: Remove minio init and quiet handling (#3739) 2017-02-13 00:24:34 -08:00
Harshavardhana 22909c849e objcache: Return io.ReaderAt to avoid Seeking and Reading. (#3735) 2017-02-11 17:17:58 -08:00
Bala FA 440866d26c Move go version check into main() (#3734)
Previously the check was done in init().  This patch moves into main
for unit testable friendly function.
2017-02-11 01:27:27 -08:00
Krishna Srinivas 25b936c369 browser: Implement infinite scrolling for object listing. (#3720)
fixes #2831
2017-02-10 22:54:42 -08:00
Nitish Tiwari 8f66cfa316 Fix https://github.com/minio/minio/issues/3453 (#3733) 2017-02-10 22:53:17 -08:00
Bala FA 8912b6bf3b trie: new package (#3729)
This implements a simple trie tree for minio server/tools.
This package borrows idea from
https://godoc.org/golang.org/x/text/internal/triegen.
2017-02-10 11:51:41 -08:00
Karthic Rao a799351926 Fix macOS builds (#3728) 2017-02-10 00:37:22 -08:00
Harshavardhana 9df01035da Remove XL references in public docs to Erasure. (#3725)
Ref #3722
2017-02-09 23:26:44 -08:00
Krishnan Parthasarathi 1ad96ee09f Handle uptime for single-node instances differnt from distributed setup (#3726) 2017-02-09 20:38:14 -08:00
Harshavardhana 1b4bb94ac4 config: setter/getter for Notifier and Logger into its own struct. (#3721)
This is an attempt cleanup code and keep the top level config
functions simpler and easy to understand where as move the
notifier related code and logger setter/getter methods as part
of their own struct.

Locks are now held properly not globally by configMutex, but
instead as private variables.

Final fix for #3700
2017-02-09 15:20:54 -08:00
Andreas Auernhammer f38222c0cc update the blake2b implementation (#3724)
Fixes a performance bug caused by SSE-AVX register savings on amd64.
2017-02-09 15:01:00 -08:00
Anis Elleuch c9b1468c3b Presigend Post: Error out when File is not found (#3723)
Follow S3 behavior when no File is sent in the presigned
post request form.
2017-02-09 12:37:32 -08:00
Krishna Srinivas 6800902b43 web-handlers: Implement API to download files as a zip file. (#3715) 2017-02-08 23:39:08 -08:00
Krishnan Parthasarathi e5773e11c6 Make minio server compile on OpenBSD, NetBSD, Solaris (#3719) 2017-02-08 22:27:35 -08:00
Nitish Tiwari 0c7694894b Added spaces for formatting (#3717) 2017-02-08 00:51:08 -08:00
Krishnan Parthasarathi ce9aa2f2b2 Add uptime to ServiceStatus (#3690) 2017-02-08 00:13:02 -08:00
Nitish Tiwari 7547f3c8a3 Added Webhook notification details (#3706) 2017-02-07 23:19:32 -08:00
Harshavardhana 31dff87903 Honor envs properly for access and secret key. (#3703)
Also changes the behavior of `secretKeyHash` which is
not necessary to be sent over the network, each node
has its own secretKeyHash to validate.

Fixes #3696
Partial(fix) #3700 (More changes needed with some code cleanup)
2017-02-07 12:51:43 -08:00
Anis Elleuch fd72c21e0e tests: Reduce two functions work to gain test time (#3712)
TestListObjectsHeal and TestFSShutdown takes around 3 min,
this PR reduces the number of created test objects
2017-02-07 12:51:23 -08:00
Harshavardhana 310bf5bd36 auth/rpc: Make auth rpc client retry configurable. (#3695)
Currently the auth rpc client defaults to to a maximum
cap of 30seconds timeout. Make this to be configurable
by the caller of authRPCClient during initialization, if no
such config is provided then default to 30 seconds.
2017-02-07 02:16:29 -08:00
Harshavardhana a170e44689 Let hijacker and flusher interfaces to be reflected. (#3709)
Ideally here if the interface is not found it would
fail the server, as it should be because without these
we can't even have a working server in the first place.

Just like how it fails in master invariably inside Go
net/http code path.

Fixes #3708
2017-02-06 23:52:47 -08:00
Anis Elleuch 70e70446bb signv4: Read always returns EOF when stream ends (#3692)
When EOF is reached, further calls of Read() doesn't return io.EOF
but continue to work as it expects to have more data, this PR fixes
the behavior
2017-02-06 14:19:27 -08:00
Krishna Srinivas 45d9cfa0c5 signature-v4: stringToSign and signingKey should use Scope's date. (#3688)
fixes #3676
2017-02-06 13:09:09 -08:00
Anis Elleuch 93fd269329 stats: Add network and http statisics (#3686)
Network: total bytes of incoming and outgoing server's data
by taking advantage of our ConnMux Read/Write wrapping

HTTP: total number of different http verbs passed in http
requests and different status codes passed in http responses.
This is counted in a new http handler.
2017-02-06 09:29:53 -08:00
Harshavardhana 6717a0b68c Add consistency guarantees 2017-02-05 18:31:48 -08:00
Harshavardhana 533338bdeb all/windows: Be case in-sensitive about pattern matching. (#3682)
Resource strings and paths are case insensitive on windows
deployments but if user happens to use upper case instead of
lower case for certain configuration params like bucket
policies and bucket notification config. We might not honor
them which leads to a wrong behavior on windows.

This is windows only behavior, for all other platforms case
is still kept sensitive.
2017-02-03 23:27:50 -08:00
Anis Elleuch b6ebf2aba8 server-mux: Simplify graceful shutdown behavior (#3681)
`*http.Server` is no more used, doing some cleanup.
2017-02-03 22:53:30 -08:00
Anis Elleuch ed4fcb63f7 Require content-length in POST & Upload requests (#3671)
Avoid passing size = -1 to PutObject API by requiring content-length
header in POST request (as AWS S3 does) and in Upload web handler.
Post handler is modified to completely store multipart file to know
its size before sending it to PutObject().
2017-02-02 10:45:00 -08:00
Harshavardhana 4b4cb07fb6 Update browser assets with new changes. 2017-02-02 03:01:49 -08:00
Harshavardhana fc446e0b81 docs: Add browser doc for running dev in custom port (#3675)
Fixes #3674
2017-02-01 13:12:13 -08:00
Anis Elleuch f612a7dd85 madmin: Fix a typo in Locks duration query name (#3673) 2017-02-01 11:46:49 -08:00
Krishnan Parthasarathi 0472e5c1e1 Change query param name to duration in list/clear locks API (#3664)
Following is a sample list lock API request schematic,

  /?lock&bucket=mybucket&prefix=myprefix&duration=holdDuration
  x-minio-operation: list

The response would contain the list of locks held on mybucket matching
myprefix for a duration longer than holdDuration.
2017-02-01 11:17:30 -08:00
Harshavardhana 6a6c930f5b xl: Abort multipart upload should honor quorum properly. (#3670)
Current implementation didn't honor quorum properly and didn't
handle the errors generated properly. This patch addresses that
and also moves common code `cleanupMultipartUploads` into xl
specific private function.

Fixes #3665
2017-02-01 11:16:17 -08:00
Karthic Rao 35ca3e5d9b Fix unresponsiveness of doneCh due to Sleep call. (#3667) 2017-02-01 09:17:32 -08:00
Harshavardhana a9ab01731f docs: Remove reference word from notification title 2017-01-31 18:17:31 -08:00
Harshavardhana cd80e6df29 docs: Move the notifications into docs/bucket
Cleanup some formatting issues.
2017-01-31 18:07:39 -08:00
koolhead17 4dea4f3b89 docs: added event-notification back end doc. (#3510) 2017-01-31 17:04:36 -08:00
Harshavardhana 1b30a3be2b xl/utils: getPartSizeFromIdx should return error. (#3669) 2017-01-31 15:34:49 -08:00
Krishna Srinivas f7f103725b browser: Make logo and "Minio Browser" text non-clickable. (#3668)
fixes #3601
2017-01-31 13:44:07 -08:00
Harshavardhana 77a192a7b5 Implement CopyObjectPart API (#3663)
This API is implemented to allow copying data from an
existing source object to an ongoing multipart operation

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html

Fixes #3662
2017-01-31 09:38:34 -08:00
Harshavardhana cb48517a78 Update browser assets with new changes. 2017-01-30 16:38:49 -08:00
Anis Elleuch e9394dc22d xl PutObject: Split object into parts (#3651)
For faster time-to-first-byte when we try to download a big object
2017-01-30 15:44:42 -08:00
Krishna Srinivas 46743c7918 browser: Component attr names should not clash with redux state keys. (#3656)
fixes #270
2017-01-30 14:25:55 -08:00
Krishna Srinivas b288eaddb3 xl: bit-rot algo was not set in get-object. (#3652)
fixes #3650
2017-01-30 14:25:28 -08:00
Krishnan Parthasarathi 2665aba555 Fail PutBucketPolicy if conditions are incompatible with actions. (#3659) 2017-01-30 09:20:16 -08:00
Krishnan Parthasarathi 9b6bcb30d9 Fix handling of StringNotEquals condition operator (#3660) 2017-01-30 09:18:10 -08:00
Harshavardhana ac9ba13c19 build: Increase the travis build test timeout to 15mins 2017-01-30 01:39:53 -08:00
Krishnan Parthasarathi 864b8795aa heal: Should delete stale object parts before healing (#3649) 2017-01-30 00:45:56 -08:00
Bala FA cc1575f944 fix: do port availability check only on macOS. (#3654)
On macOS, if a process already listens on 127.0.0.1:PORT, net.Listen() falls back
to IPv6 address ie minio will start listening on IPv6 address whereas another
(non-)minio process is listening on IPv4 of given port.
To avoid this error sutiation we check for port availability only for macOS.

Note: checkPortAvailability() tries to listen on given port and closes it.
It is possible to have a disconnected client in this tiny window of time.
2017-01-30 00:44:36 -08:00
Krishnan Parthasarathi b408d0e87d Add aws:Referer condition key support. (#3641)
This change implements bucket policy enhancements required to restrict access based on HTTP referer.
See https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-4 for more information.

Fixes #3540
2017-01-29 19:45:11 -08:00
Nitish Tiwari 69b81af93e Fix Minio quick start guide URL (#3655) 2017-01-29 07:21:30 -08:00
Harshavardhana cf558ff6d1 docs: Fix TLS doc to be docs.minio.io compatible 2017-01-28 22:46:29 -08:00
Harshavardhana 5af7bd6a01 docs: Fix a typo in TLS doc description 2017-01-28 22:41:52 -08:00
Harshavardhana 73f4f29110 docs: Restructure docs, move the files to their relevant location. (#3648)
Also combines windows TLS docs with single doc with Linux.
2017-01-28 19:45:30 -08:00
Harshavardhana ccd949d8ca Add -v for race tests to avoid travis and appveyor timeouts. (#3647)
This is added so that CI's don't wrong fully timeout on
little longer running tests.
2017-01-27 12:31:02 -08:00
Anis Elleuch e1bc99e4fe xl: Fix GET of an empty multiparted object (#3646)
GetObject returns unsatisfied range error when we try to download an object
uploaded using multipart mechanism.
2017-01-27 10:51:02 -08:00
Alex Ellis 02194ee3c6 Fix issue with webhook events not closing Response body, leaving connections open with remote server (#3645) 2017-01-27 09:50:08 -08:00
Harshavardhana 85f2b74cfd jwt: Cache the bcrypt password hash. (#3526)
Creds don't require secretKeyHash to be calculated
everytime, cache it instead and re-use.

This is an optimization for bcrypt.

Relevant results from the benchmark done locally, negative
value means improvement in this scenario.

```
benchmark                       old ns/op     new ns/op     delta
BenchmarkAuthenticateNode-4     160590992     80125647      -50.11%
BenchmarkAuthenticateWeb-4      160556692     80432144      -49.90%

benchmark                       old allocs     new allocs     delta
BenchmarkAuthenticateNode-4     87             75             -13.79%
BenchmarkAuthenticateWeb-4      87             75             -13.79%

benchmark                       old bytes     new bytes     delta
BenchmarkAuthenticateNode-4     15222         9785          -35.72%
BenchmarkAuthenticateWeb-4      15222         9785          -35.72%
```
2017-01-26 16:51:51 -08:00
Krishna Srinivas 152cdf1c05 fs: Move traceError() to lower functions where possible. (#3633) 2017-01-26 15:40:10 -08:00
Krishna Srinivas 17dd1c19df cleanup: refactor common code between FS and XL listDirFactory. (#3639) 2017-01-26 15:39:22 -08:00
Anis Elleuch 8e49a3d047 Simplify running cmd.Main() for external tests (#3636)
An external test that runs cmd.Main() has a difficulty to set cmd arguments
and MINIO_{ACCESS,SECRET}_KEY values, this commit changes a little the current
behavior in a way that helps external tests.
2017-01-26 15:22:41 -08:00
Krishna Srinivas cccf77229d cleanup: Move code in *-multipart-common.go to *-multipart.go (#3638)
The code in *-multipart-common.go is not common anymore.
2017-01-26 12:51:12 -08:00
Harshavardhana dafdc74605 fs: if fs.json is empty ignore it while reading metadata. (#3634)
This is needed so that we don't send wrong errors
on previously failed PutObject() which would have
left a stale `fs.json` entry.
2017-01-26 10:19:07 -08:00
Anis Elleuch c753b5dfeb madmin: All APIs return ErrorResponse error type (#3632)
Avoid returning errors.New("Got HTTP Status") in all APIs
2017-01-25 16:01:29 -08:00
Krishna Srinivas 82373e3d50 fs: cleanup - do not cache size of metafiles (#3630)
* Remove Size() method and size field from lock.LockedFile
* WriteTo method of fsMeta and uploadsV1 now takes concrete type *lock.LockedFile
2017-01-25 12:29:06 -08:00
Anis Elleuch 76b8abcd8e madmin: Fix args order in listObjectsHeal() (#3629)
The order of marker and delimiter and in listObjectsHeal() internal function
are switched. That will give wrong result in case of a non recursive objects
heal list.
2017-01-25 09:58:49 -08:00
Harshavardhana dbb5408906 Add ARM64 build binary for README.md
Fixes #3626
2017-01-24 20:28:54 -08:00
417 changed files with 37019 additions and 12654 deletions
+18 -2
View File
@@ -1,6 +1,9 @@
go_import_path: github.com/minio/minio
sudo: required
services:
- docker
dist: trusty
language: go
@@ -13,12 +16,25 @@ env:
- ARCH=i686
script:
## Run all the tests
- make
- make test GOFLAGS="-race"
- make test GOFLAGS="-timeout 15m -race -v"
- make coverage
# Refer https://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/
# push image
- >
if [ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && [ "$ARCH" == "x86_64" ]; then
docker run --rm --privileged multiarch/qemu-user-static:register --reset
docker build -t minio/minio:edge-armhf . -f Dockerfile.armhf
docker build -t minio/minio:edge-aarch64 . -f Dockerfile.aarch64
docker login -u="$DOCKER_USER" -p="$DOCKER_PASS"
docker push minio/minio:edge-armhf
docker push minio/minio:edge-aarch64
fi
after_success:
- bash <(curl -s https://codecov.io/bash)
go:
- 1.7.4
- 1.7.5
+12 -10
View File
@@ -1,16 +1,18 @@
FROM golang:1.7-alpine
FROM alpine:3.5
WORKDIR /go/src/app
ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
COPY . /go/src/app
WORKDIR /go/src/github.com/minio/
RUN \
apk add --no-cache git && \
go-wrapper download && \
go-wrapper install -ldflags "-X github.com/minio/minio/cmd.Version=2017-01-25T03:14:52Z -X github.com/minio/minio/cmd.ReleaseTag=RELEASE.2017-01-25T03-14-52Z -X github.com/minio/minio/cmd.CommitID=f8e4700a11065967242b8857045ce7dee607722d" && \
mkdir -p /export/docker && \
rm -rf /go/pkg /go/src && \
apk del git
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps git go musl-dev && \
go get -v -d github.com/minio/minio && \
cd /go/src/github.com/minio/minio && \
go install -v -ldflags "-X github.com/minio/minio/cmd.Version=2017-04-29T00:40:27Z -X github.com/minio/minio/cmd.ReleaseTag=RELEASE.2017-04-29T00-40-27Z -X github.com/minio/minio/cmd.CommitID=eb50175ad911d496bf583a599de89547f0c9be89" && \
rm -rf /go/pkg /go/src /usr/local/go && apk del .build-deps
EXPOSE 9000
ENTRYPOINT ["minio"]
+19
View File
@@ -0,0 +1,19 @@
FROM resin/aarch64-alpine:3.5
ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
WORKDIR /go/src/github.com/minio/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps git go musl-dev && \
go get -v -d github.com/minio/minio && \
cd /go/src/github.com/minio/minio && \
go install -v -ldflags "-X github.com/minio/minio/cmd.Version=2017-03-16T21:50:32Z -X github.com/minio/minio/cmd.ReleaseTag=RELEASE.2017-03-16T21-50-32Z -X github.com/minio/minio/cmd.CommitID=5311eb22fd681a8cd4a46e2a872d46c2352c64e8" && \
rm -rf /go/pkg /go/src /usr/local/go && apk del .build-deps
EXPOSE 9000
ENTRYPOINT ["minio"]
VOLUME ["/export"]
+19
View File
@@ -0,0 +1,19 @@
FROM resin/armhf-alpine:3.5
ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
WORKDIR /go/src/github.com/minio/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps git go musl-dev && \
go get -v -d github.com/minio/minio && \
cd /go/src/github.com/minio/minio && \
go install -v -ldflags "-X github.com/minio/minio/cmd.Version=2017-03-16T21:50:32Z -X github.com/minio/minio/cmd.ReleaseTag=RELEASE.2017-03-16T21-50-32Z -X github.com/minio/minio/cmd.CommitID=5311eb22fd681a8cd4a46e2a872d46c2352c64e8" && \
rm -rf /go/pkg /go/src /usr/local/go && apk del .build-deps
EXPOSE 9000
ENTRYPOINT ["minio"]
VOLUME ["/export"]
+4 -5
View File
@@ -1,7 +1,7 @@
LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
PWD := $(shell pwd)
GOPATH := $(shell go env GOPATH)
BUILD_LDFLAGS := '$(LDFLAGS)'
BUILD_LDFLAGS := '$(LDFLAGS) -s -w'
TAG := latest
HOST ?= $(shell uname)
@@ -71,10 +71,8 @@ verifiers: vet fmt lint cyclo spelling
vet:
@echo "Running $@:"
@go tool vet -all ./cmd
@go tool vet -all ./pkg
@go tool vet -shadow=true ./cmd
@go tool vet -shadow=true ./pkg
@go vet github.com/minio/minio/cmd/...
@go vet github.com/minio/minio/pkg/...
fmt:
@echo "Running $@:"
@@ -107,6 +105,7 @@ spelling:
test: build
@echo "Running all minio testing:"
@go test $(GOFLAGS) .
@go test $(GOFLAGS) github.com/minio/minio/cmd...
@go test $(GOFLAGS) github.com/minio/minio/pkg...
+5 -3
View File
@@ -18,19 +18,19 @@ docker run -p 9000:9000 minio/minio:edge server /export
```
Please visit Minio Docker quickstart guide for more [here](https://docs.minio.io/docs/minio-docker-quickstart-guide)
## OS X
## macOS
### Homebrew
Install minio packages using [Homebrew](http://brew.sh/)
```sh
brew install minio
brew install minio/stable/minio
minio server ~/Photos
```
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|Apple OS X|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio|
|Apple macOS|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio |
```sh
chmod 755 minio
./minio server ~/Photos
@@ -43,6 +43,8 @@ chmod 755 minio
|GNU/Linux|64-bit Intel|https://dl.minio.io/server/minio/release/linux-amd64/minio|
||32-bit Intel|https://dl.minio.io/server/minio/release/linux-386/minio|
||32-bit ARM|https://dl.minio.io/server/minio/release/linux-arm/minio|
||64-bit ARM|https://dl.minio.io/server/minio/release/linux-arm64/minio|
||32-bit ARMv6|https://dl.minio.io/server/minio/release/linux-arm6vl/minio|
```sh
chmod +x minio
./minio server ~/Photos
+2
View File
@@ -11,6 +11,8 @@ Minio是一个非常轻量的服务,可以很简单的和其他应用的结合
|GNU/Linux|64-bit Intel|https://dl.minio.io/server/minio/release/linux-amd64/minio|
||32-bit Intel|https://dl.minio.io/server/minio/release/linux-386/minio|
||32-bit ARM|https://dl.minio.io/server/minio/release/linux-arm/minio|
||64-bit ARM|https://dl.minio.io/server/minio/release/linux-arm64/minio|
||32-bit ARMv6|https://dl.minio.io/server/minio/release/linux-arm6vl/minio|
|Apple OS X|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio|
|Microsoft Windows|64-bit|https://dl.minio.io/server/minio/release/windows-amd64/minio.exe|
||32-bit|https://dl.minio.io/server/minio/release/windows-386/minio.exe|
+4 -3
View File
@@ -11,11 +11,12 @@ clone_folder: c:\gopath\src\github.com\minio\minio
# Environment variables
environment:
GOROOT: c:\go17
GOPATH: c:\gopath
# scripts that run after cloning repository
install:
- set PATH=%GOPATH%\bin;c:\go\bin;%PATH%
- set PATH=%GOPATH%\bin;c:\go17\bin;%PATH%
- go version
- go env
- python --version
@@ -35,8 +36,8 @@ test_script:
# Unit tests
- ps: Add-AppveyorTest "Unit Tests" -Outcome Running
- mkdir build\coverage
- go test -timeout 15m -v -race github.com/minio/minio/cmd...
- go test -v -race github.com/minio/minio/pkg...
- go test -timeout 17m -race github.com/minio/minio/cmd...
- go test -race github.com/minio/minio/pkg...
- go test -coverprofile=build\coverage\coverage.txt -covermode=atomic github.com/minio/minio/cmd
- ps: Update-AppveyorTest "Unit Tests" -Outcome Passed
+46 -11
View File
@@ -4,34 +4,69 @@
## Installation
### Install yarn:
### Install yarn
```sh
$ curl -o- -L https://yarnpkg.com/install.sh | bash
$ yarn
curl -o- -L https://yarnpkg.com/install.sh | bash
yarn
```
### Install `go-bindata` and `go-bindata-assetfs`.
### Install `go-bindata` and `go-bindata-assetfs`
If you do not have a working Golang environment, please follow [Install Golang](https://docs.minio.io/docs/how-to-install-golang)
```sh
$ go get github.com/jteeuwen/go-bindata/...
$ go get github.com/elazarl/go-bindata-assetfs/...
go get github.com/jteeuwen/go-bindata/...
go get github.com/elazarl/go-bindata-assetfs/...
```
## Generating Assets.
## Generating Assets
### Generate ui-assets.go
```sh
$ yarn release
yarn release
```
This generates ui-assets.go in the current direcotry. Now do `make` in the parent directory to build the minio binary with the newly generated ui-assets.go
### Run Minio Browser with live reload.
This generates ui-assets.go in the current direcotry. Now do `make` in the parent directory to build the minio binary with the newly generated ``ui-assets.go``
### Run Minio Browser with live reload
```sh
$ yarn dev
yarn dev
```
Open [http://localhost:8080/minio/](http://localhost:8080/minio/) in your browser to play with the application
### Run Minio Browser with live reload on custom port
Edit `browser/webpack.config.js`
```diff
diff --git a/browser/webpack.config.js b/browser/webpack.config.js
index 3ccdaba..9496c56 100644
--- a/browser/webpack.config.js
+++ b/browser/webpack.config.js
@@ -58,6 +58,7 @@ var exports = {
historyApiFallback: {
index: '/minio/'
},
+ port: 8888,
proxy: {
'/minio/webrpc': {
target: 'http://localhost:9000',
@@ -97,7 +98,7 @@ var exports = {
if (process.env.NODE_ENV === 'dev') {
exports.entry = [
'webpack/hot/dev-server',
- 'webpack-dev-server/client?http://localhost:8080',
+ 'webpack-dev-server/client?http://localhost:8888',
path.resolve(__dirname, 'app/index.js')
]
}
```
```sh
yarn dev
```
Open [http://localhost:8888/minio/](http://localhost:8888/minio/) in your browser to play with the application
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+118 -22
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -14,13 +14,9 @@
* limitations under the License.
*/
import url from 'url'
import Moment from 'moment'
import browserHistory from 'react-router/lib/browserHistory'
import web from './web'
import * as utils from './utils'
import storage from 'local-storage-fallback'
import { minioBrowserPrefix } from './constants'
export const SET_WEB = 'SET_WEB'
@@ -28,7 +24,6 @@ export const SET_CURRENT_BUCKET = 'SET_CURRENT_BUCKET'
export const SET_CURRENT_PATH = 'SET_CURRENT_PATH'
export const SET_BUCKETS = 'SET_BUCKETS'
export const ADD_BUCKET = 'ADD_BUCKET'
export const ADD_OBJECT = 'ADD_OBJECT'
export const SET_VISIBLE_BUCKETS = 'SET_VISIBLE_BUCKETS'
export const SET_OBJECTS = 'SET_OBJECTS'
export const SET_STORAGE_INFO = 'SET_STORAGE_INFO'
@@ -56,6 +51,10 @@ export const SET_POLICIES = 'SET_POLICIES'
export const SET_SHARE_OBJECT = 'SET_SHARE_OBJECT'
export const DELETE_CONFIRMATION = 'DELETE_CONFIRMATION'
export const SET_PREFIX_WRITABLE = 'SET_PREFIX_WRITABLE'
export const REMOVE_OBJECT = 'REMOVE_OBJECT'
export const CHECKED_OBJECTS_ADD = 'CHECKED_OBJECTS_ADD'
export const CHECKED_OBJECTS_REMOVE = 'CHECKED_OBJECTS_REMOVE'
export const CHECKED_OBJECTS_RESET = 'CHECKED_OBJECTS_RESET'
export const showDeleteConfirmation = (object) => {
return {
@@ -77,11 +76,12 @@ export const hideDeleteConfirmation = () => {
}
}
export const showShareObject = url => {
export const showShareObject = (object, url) => {
return {
type: SET_SHARE_OBJECT,
shareObject: {
url: url,
object,
url,
show: true
}
}
@@ -97,15 +97,17 @@ export const hideShareObject = () => {
}
}
export const shareObject = (object, expiry) => (dispatch, getState) => {
export const shareObject = (object, days, hours, minutes) => (dispatch, getState) => {
const {currentBucket, web} = getState()
let host = location.host
let bucket = currentBucket
if (!web.LoggedIn()) {
dispatch(showShareObject(`${host}/${bucket}/${object}`))
dispatch(showShareObject(object, `${host}/${bucket}/${object}`))
return
}
let expiry = days * 24 * 60 * 60 + hours * 60 * 60 + minutes * 60
web.PresignedGet({
host,
bucket,
@@ -113,7 +115,11 @@ export const shareObject = (object, expiry) => (dispatch, getState) => {
expiry
})
.then(obj => {
dispatch(showShareObject(obj.url))
dispatch(showShareObject(object, obj.url))
dispatch(showAlert({
type: 'success',
message: `Object shared. Expires in ${days} days ${hours} hours ${minutes} minutes.`
}))
})
.catch(err => {
dispatch(showAlert({
@@ -206,6 +212,13 @@ export const showAlert = alert => {
}
}
export const removeObject = object => {
return {
type: REMOVE_OBJECT,
object
}
}
export const setSidebarStatus = (status) => {
return {
type: SET_SIDEBAR_STATUS,
@@ -227,10 +240,12 @@ export const setVisibleBuckets = visibleBuckets => {
}
}
export const setObjects = (objects) => {
export const setObjects = (objects, marker, istruncated) => {
return {
type: SET_OBJECTS,
objects
objects,
marker,
istruncated
}
}
@@ -284,22 +299,63 @@ export const selectBucket = (newCurrentBucket, prefix) => {
}
}
export const selectPrefix = prefix => {
export const listObjects = () => {
return (dispatch, getState) => {
const {currentBucket, web} = getState()
dispatch(setLoadPath(prefix))
const {currentBucket, currentPath, marker, objects, istruncated, web} = getState()
if (!istruncated) return
web.ListObjects({
bucketName: currentBucket,
prefix
prefix: currentPath,
marker: marker
})
.then(res => {
let objects = res.objects
if (!objects)
objects = []
objects = objects.map(object => {
object.name = object.name.replace(`${currentPath}`, '');
return object
})
dispatch(setObjects(objects, res.nextmarker, res.istruncated))
dispatch(setPrefixWritable(res.writable))
dispatch(setLoadBucket(''))
dispatch(setLoadPath(''))
})
.catch(err => {
dispatch(showAlert({
type: 'danger',
message: err.message
}))
dispatch(setLoadBucket(''))
dispatch(setLoadPath(''))
// Use browserHistory.replace instead of push so that browser back button works fine.
browserHistory.replace(`${minioBrowserPrefix}/login`)
})
}
}
export const selectPrefix = prefix => {
return (dispatch, getState) => {
const {currentBucket, web} = getState()
dispatch(setObjects([], "", false))
dispatch(setLoadPath(prefix))
web.ListObjects({
bucketName: currentBucket,
prefix,
marker: ""
})
.then(res => {
let objects = res.objects
if (!objects)
objects = []
objects = objects.map(object => {
object.name = object.name.replace(`${prefix}`, '');
return object
})
dispatch(setObjects(
utils.sortObjectsByName(objects.map(object => {
object.name = object.name.replace(`${prefix}`, ''); return object
}))
objects,
res.nextmarker,
res.istruncated
))
dispatch(setPrefixWritable(res.writable))
dispatch(setSortNameOrder(false))
@@ -314,8 +370,8 @@ export const selectPrefix = prefix => {
}))
dispatch(setLoadBucket(''))
dispatch(setLoadPath(''))
// Use browserHistory.replace instead of push so that browser back button works fine.
browserHistory.replace(`${minioBrowserPrefix}/login`)
// Use browserHistory.replace instead of push so that browser back button works fine.
browserHistory.replace(`${minioBrowserPrefix}/login`)
})
}
}
@@ -359,6 +415,25 @@ export const setLoginError = () => {
}
}
export const downloadSelected = (url, req, xhr) => {
return (dispatch) => {
xhr.open('POST', url, true)
xhr.responseType = 'blob'
xhr.onload = function(e) {
if (this.status == 200) {
dispatch(checkedObjectsReset())
var blob = new Blob([this.response], {
type: 'application/zip'
})
var blobUrl = window.URL.createObjectURL(blob);
window.location = blobUrl
}
};
xhr.send(JSON.stringify(req));
}
}
export const uploadFile = (file, xhr) => {
return (dispatch, getState) => {
const {currentBucket, currentPath} = getState()
@@ -512,3 +587,24 @@ export const setPolicies = (policies) => {
policies
}
}
export const checkedObjectsAdd = (objectName) => {
return {
type: CHECKED_OBJECTS_ADD,
objectName
}
}
export const checkedObjectsRemove = (objectName) => {
return {
type: CHECKED_OBJECTS_REMOVE,
objectName
}
}
export const checkedObjectsReset = (objectName) => {
return {
type: CHECKED_OBJECTS_RESET,
objectName
}
}
+98 -47
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -27,7 +27,6 @@ import OverlayTrigger from 'react-bootstrap/lib/OverlayTrigger'
import Tooltip from 'react-bootstrap/lib/Tooltip'
import Dropdown from 'react-bootstrap/lib/Dropdown'
import MenuItem from 'react-bootstrap/lib/MenuItem'
import InputGroup from '../components/InputGroup'
import Dropzone from '../components/Dropzone'
import ObjectsList from '../components/ObjectsList'
@@ -47,6 +46,7 @@ import * as mime from '../mime'
import { minioBrowserPrefix } from '../constants'
import CopyToClipboard from 'react-copy-to-clipboard'
import storage from 'local-storage-fallback'
import InfiniteScroll from 'react-infinite-scroller';
export default class Browse extends React.Component {
componentDidMount() {
@@ -110,9 +110,6 @@ export default class Browse extends React.Component {
if (!decPathname.endsWith('/'))
decPathname += '/'
if (decPathname === minioBrowserPrefix + '/') {
dispatch(actions.setCurrentBucket(''))
dispatch(actions.setCurrentPath(''))
dispatch(actions.setObjects([]))
return
}
let obj = utils.pathSlice(decPathname)
@@ -140,6 +137,11 @@ export default class Browse extends React.Component {
this.props.dispatch(actions.setVisibleBuckets(buckets.filter(bucket => bucket.indexOf(e.target.value) > -1)))
}
listObjects() {
const {dispatch} = this.props
dispatch(actions.listObjects())
}
selectPrefix(e, prefix) {
e.preventDefault()
const {dispatch, currentPath, web, currentBucket} = this.props
@@ -224,14 +226,24 @@ export default class Browse extends React.Component {
}
removeObject() {
const {web, dispatch, currentPath, currentBucket, deleteConfirmation} = this.props
const {web, dispatch, currentPath, currentBucket, deleteConfirmation, checkedObjects} = this.props
let objects = checkedObjects.length > 0 ? checkedObjects : [deleteConfirmation.object]
web.RemoveObject({
bucketName: currentBucket,
objectName: deleteConfirmation.object
bucketname: currentBucket,
objects: objects
})
.then(() => {
this.hideDeleteConfirmation()
dispatch(actions.selectPrefix(currentPath))
if (checkedObjects.length > 0) {
for (let i = 0; i < checkedObjects.length; i++) {
dispatch(actions.removeObject(checkedObjects[i].replace(currentPath, '')))
}
dispatch(actions.checkedObjectsReset())
} else {
let delObject = deleteConfirmation.object.replace(currentPath, '')
dispatch(actions.removeObject(delObject))
}
})
.catch(e => dispatch(actions.showAlert({
type: 'danger',
@@ -259,7 +271,8 @@ export default class Browse extends React.Component {
shareObject(e, object) {
e.preventDefault()
const {dispatch} = this.props
dispatch(actions.shareObject(object))
// let expiry = 5 * 24 * 60 * 60 // 5 days expiry by default
dispatch(actions.shareObject(object, 5, 0, 0))
}
hideShareObjectModal() {
@@ -296,11 +309,6 @@ export default class Browse extends React.Component {
browserHistory.push(`${minioBrowserPrefix}/login`)
}
landingPage(e) {
e.preventDefault()
this.props.dispatch(actions.selectBucket(this.props.buckets[0]))
}
fullScreen(e) {
e.preventDefault()
let el = document.documentElement
@@ -356,26 +364,53 @@ export default class Browse extends React.Component {
this.refs.copyTextInput.select()
}
handleExpireValue(targetInput, inc) {
handleExpireValue(targetInput, inc, object) {
inc === -1 ? this.refs[targetInput].stepDown(1) : this.refs[targetInput].stepUp(1)
if (this.refs.expireDays.value == 7) {
this.refs.expireHours.value = 0
this.refs.expireMins.value = 0
}
if (this.refs.expireDays.value + this.refs.expireHours.value + this.refs.expireMins.value == 0) {
this.refs.expireDays.value = 7
}
const {dispatch} = this.props
dispatch(actions.shareObject(object, this.refs.expireDays.value, this.refs.expireHours.value, this.refs.expireMins.value))
}
checkObject(e, objectName) {
const {dispatch} = this.props
e.target.checked ? dispatch(actions.checkedObjectsAdd(objectName)) : dispatch(actions.checkedObjectsRemove(objectName))
}
downloadSelected() {
const {dispatch} = this.props
let req = {
bucketName: this.props.currentBucket,
objects: this.props.checkedObjects,
prefix: this.props.currentPath
}
let requestUrl = location.origin + "/minio/zip?token=" + localStorage.token
this.xhr = new XMLHttpRequest()
dispatch(actions.downloadSelected(requestUrl, req, this.xhr))
}
clearSelected() {
const {dispatch} = this.props
dispatch(actions.checkedObjectsReset())
}
render() {
const {total, free} = this.props.storageInfo
const {showMakeBucketModal, alert, sortNameOrder, sortSizeOrder, sortDateOrder, showAbout, showBucketPolicy} = this.props
const {showMakeBucketModal, alert, sortNameOrder, sortSizeOrder, sortDateOrder, showAbout, showBucketPolicy, checkedObjects} = this.props
const {version, memory, platform, runtime} = this.props.serverInfo
const {sidebarStatus} = this.props
const {showSettings} = this.props
const {policies, currentBucket, currentPath} = this.props
const {deleteConfirmation} = this.props
const {shareObject} = this.props
const {web, prefixWritable} = this.props
const {web, prefixWritable, istruncated} = this.props
// Don't always show the SettingsModal. This is done here instead of in
// SettingsModal.js so as to allow for #componentWillMount to handle
@@ -414,10 +449,10 @@ export default class Browse extends React.Component {
let freePercent = free * 100 / total
if (web.LoggedIn()) {
browserDropdownButton = <BrowserDropdown fullScreen={ this.fullScreen.bind(this) }
showAbout={ this.showAbout.bind(this) }
showSettings={ this.showSettings.bind(this) }
logout={ this.logout.bind(this) } />
browserDropdownButton = <BrowserDropdown fullScreenFunc={ this.fullScreen.bind(this) }
aboutFunc={ this.showAbout.bind(this) }
settingsFunc={ this.showSettings.bind(this) }
logoutFunc={ this.logout.bind(this) } />
} else {
loginButton = <a className='btn btn-danger' href='/minio/login'>Login</a>
}
@@ -438,7 +473,6 @@ export default class Browse extends React.Component {
</li>
</ul>
</div>
}
let createButton = ''
@@ -481,7 +515,6 @@ export default class Browse extends React.Component {
</OverlayTrigger>
</Dropdown.Menu>
</Dropdown>
}
return (
@@ -489,12 +522,19 @@ export default class Browse extends React.Component {
'file-explorer': true,
'toggled': sidebarStatus
}) }>
<SideBar landingPage={ this.landingPage.bind(this) }
searchBuckets={ this.searchBuckets.bind(this) }
<SideBar searchBuckets={ this.searchBuckets.bind(this) }
selectBucket={ this.selectBucket.bind(this) }
clickOutside={ this.hideSidebar.bind(this) }
showPolicy={ this.showBucketPolicy.bind(this) } />
<div className="fe-body">
<div className={ 'list-actions' + (classNames({
' list-actions-toggled': checkedObjects.length > 0
})) }>
<span className="la-label"><i className="fa fa-check-circle" /> { checkedObjects.length } Objects selected</span>
<span className="la-actions pull-right"><button onClick={ this.downloadSelected.bind(this) }> Download all as zip </button></span>
<span className="la-actions pull-right"><button onClick={ this.showDeleteConfirmation.bind(this) }> Delete selected </button></span>
<i className="la-close fa fa-times" onClick={ this.clearSelected.bind(this) }></i>
</div>
<Dropzone>
{ alertBox }
<header className="fe-header-mobile hidden-lg hidden-md">
@@ -520,7 +560,8 @@ export default class Browse extends React.Component {
</header>
<div className="feb-container">
<header className="fesl-row" data-type="folder">
<div className="fesl-item fi-name" onClick={ this.sortObjectsByName.bind(this) } data-sort="name">
<div className="fesl-item fesl-item-icon"></div>
<div className="fesl-item fesl-item-name" onClick={ this.sortObjectsByName.bind(this) } data-sort="name">
Name
<i className={ classNames({
'fesli-sort': true,
@@ -529,7 +570,7 @@ export default class Browse extends React.Component {
'fa-sort-alpha-asc': !sortNameOrder
}) } />
</div>
<div className="fesl-item fi-size" onClick={ this.sortObjectsBySize.bind(this) } data-sort="size">
<div className="fesl-item fesl-item-size" onClick={ this.sortObjectsBySize.bind(this) } data-sort="size">
Size
<i className={ classNames({
'fesli-sort': true,
@@ -538,7 +579,7 @@ export default class Browse extends React.Component {
'fa-sort-amount-asc': !sortSizeOrder
}) } />
</div>
<div className="fesl-item fi-modified" onClick={ this.sortObjectsByDate.bind(this) } data-sort="last-modified">
<div className="fesl-item fesl-item-modified" onClick={ this.sortObjectsByDate.bind(this) } data-sort="last-modified">
Last Modified
<i className={ classNames({
'fesli-sort': true,
@@ -547,14 +588,24 @@ export default class Browse extends React.Component {
'fa-sort-numeric-asc': !sortDateOrder
}) } />
</div>
<div className="fesl-item fi-actions"></div>
<div className="fesl-item fesl-item-actions"></div>
</header>
</div>
<div className="feb-container">
<ObjectsList dataType={ this.dataType.bind(this) }
selectPrefix={ this.selectPrefix.bind(this) }
showDeleteConfirmation={ this.showDeleteConfirmation.bind(this) }
shareObject={ this.shareObject.bind(this) } />
<InfiniteScroll loadMore={ this.listObjects.bind(this) }
hasMore={ istruncated }
useWindow={ true }
initialLoad={ false }>
<ObjectsList dataType={ this.dataType.bind(this) }
selectPrefix={ this.selectPrefix.bind(this) }
showDeleteConfirmation={ this.showDeleteConfirmation.bind(this) }
shareObject={ this.shareObject.bind(this) }
checkObject={ this.checkObject.bind(this) }
checkedObjectsArray={ checkedObjects } />
</InfiniteScroll>
<div className="text-center" style={ { display: (istruncated && currentBucket) ? 'block' : 'none' } }>
<span>Loading...</span>
</div>
</div>
<UploadModal />
{ createButton }
@@ -670,7 +721,7 @@ export default class Browse extends React.Component {
</label>
<div className="set-expire">
<div className="set-expire-item">
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireDays', 1) }></i>
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireDays', 1, shareObject.object) }></i>
<div className="set-expire-title">
Days
</div>
@@ -679,12 +730,12 @@ export default class Browse extends React.Component {
type="number"
min={ 0 }
max={ 7 }
defaultValue={ 0 } />
defaultValue={ 5 } />
</div>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireDays', -1) }></i>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireDays', -1, shareObject.object) }></i>
</div>
<div className="set-expire-item">
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireHours', 1) }></i>
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireHours', 1, shareObject.object) }></i>
<div className="set-expire-title">
Hours
</div>
@@ -692,30 +743,30 @@ export default class Browse extends React.Component {
<input ref="expireHours"
type="number"
min={ 0 }
max={ 24 }
max={ 23 }
defaultValue={ 0 } />
</div>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireHours', -1) }></i>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireHours', -1, shareObject.object) }></i>
</div>
<div className="set-expire-item">
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireMins', 1) }></i>
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireMins', 1, shareObject.object) }></i>
<div className="set-expire-title">
Minutes
</div>
<div className="set-expire-value">
<input ref="expireMins"
type="number"
min={ 1 }
max={ 60 }
defaultValue={ 45 } />
min={ 0 }
max={ 59 }
defaultValue={ 0 } />
</div>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireMins', -1) }></i>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireMins', -1, shareObject.object) }></i>
</div>
</div>
</div>
</div>
</ModalBody>
<div className="modal-footer">
<CopyToClipboard text={ shareObject.url } onCopy={ this.showMessage.bind(this) }>
<CopyToClipboard text={ window.location.protocol + '//' + shareObject.url } onCopy={ this.showMessage.bind(this) }>
<button className="btn btn-success">
Copy Link
</button>
+7 -7
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016, 2017 Minio, Inc.
* Minio Cloud Storage (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -18,7 +18,7 @@ import React from 'react'
import connect from 'react-redux/lib/components/connect'
import Dropdown from 'react-bootstrap/lib/Dropdown'
let BrowserDropdown = ({fullScreen, showAbout, showSettings, logout}) => {
let BrowserDropdown = ({fullScreenFunc, aboutFunc, settingsFunc, logoutFunc}) => {
return (
<li>
<Dropdown pullRight id="top-right-menu">
@@ -27,10 +27,10 @@ let BrowserDropdown = ({fullScreen, showAbout, showSettings, logout}) => {
</Dropdown.Toggle>
<Dropdown.Menu className="dropdown-menu-right">
<li>
<a target="_blank" href="https://github.com/minio/miniobrowser">Github <i className="fa fa-github"></i></a>
<a target="_blank" href="https://github.com/minio/minio">Github <i className="fa fa-github"></i></a>
</li>
<li>
<a href="" onClick={ fullScreen }>Fullscreen <i className="fa fa-expand"></i></a>
<a href="" onClick={ fullScreenFunc }>Fullscreen <i className="fa fa-expand"></i></a>
</li>
<li>
<a target="_blank" href="https://docs.minio.io/">Documentation <i className="fa fa-book"></i></a>
@@ -39,13 +39,13 @@ let BrowserDropdown = ({fullScreen, showAbout, showSettings, logout}) => {
<a target="_blank" href="https://slack.minio.io">Ask for help <i className="fa fa-question-circle"></i></a>
</li>
<li>
<a href="" onClick={ showAbout }>About <i className="fa fa-info-circle"></i></a>
<a href="" onClick={ aboutFunc }>About <i className="fa fa-info-circle"></i></a>
</li>
<li>
<a href="" onClick={ showSettings }>Settings <i className="fa fa-cog"></i></a>
<a href="" onClick={ settingsFunc }>Settings <i className="fa fa-cog"></i></a>
</li>
<li>
<a href="" onClick={ logout }>Sign Out <i className="fa fa-sign-out"></i></a>
<a href="" onClick={ logoutFunc }>Sign Out <i className="fa fa-sign-out"></i></a>
</li>
</Dropdown.Menu>
</Dropdown>
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+3 -2
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -39,11 +39,12 @@ export default class Dropzone extends React.Component {
// won't handle child elements correctly.
const style = {
height: '100%',
borderWidth: '2px',
borderWidth: '0',
borderStyle: 'dashed',
borderColor: '#fff'
}
const activeStyle = {
borderWidth: '2px',
borderColor: '#777'
}
const rejectStyle = {
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+38 -17
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -20,8 +20,7 @@ import humanize from 'humanize'
import connect from 'react-redux/lib/components/connect'
import Dropdown from 'react-bootstrap/lib/Dropdown'
let ObjectsList = ({objects, currentPath, selectPrefix, dataType, showDeleteConfirmation, shareObject, loadPath}) => {
let ObjectsList = ({objects, currentPath, selectPrefix, dataType, showDeleteConfirmation, shareObject, loadPath, checkObject, checkedObjectsArray}) => {
const list = objects.map((object, i) => {
let size = object.name.endsWith('/') ? '-' : humanize.filesize(object.size)
let lastModified = object.name.endsWith('/') ? '-' : Moment(object.lastModified).format('lll')
@@ -30,29 +29,51 @@ let ObjectsList = ({objects, currentPath, selectPrefix, dataType, showDeleteConf
let deleteButton = ''
if (web.LoggedIn())
deleteButton = <a href="" className="fiad-action" onClick={ (e) => showDeleteConfirmation(e, `${currentPath}${object.name}`) }><i className="fa fa-trash"></i></a>
if (!object.name.endsWith('/')) {
actionButtons = <Dropdown id="fia-dropdown">
<Dropdown.Toggle noCaret className="fia-toggle"></Dropdown.Toggle>
<Dropdown.Menu>
<a href="" className="fiad-action" onClick={ (e) => shareObject(e, `${currentPath}${object.name}`) }><i className="fa fa-copy"></i></a>
{ deleteButton }
</Dropdown.Menu>
</Dropdown>
if (!checkedObjectsArray.length > 0) {
if (!object.name.endsWith('/')) {
actionButtons = <Dropdown id={ "fia-dropdown-" + object.name.replace('.', '-') }>
<Dropdown.Toggle noCaret className="fia-toggle"></Dropdown.Toggle>
<Dropdown.Menu>
<a href="" className="fiad-action" onClick={ (e) => shareObject(e, `${currentPath}${object.name}`) }><i className="fa fa-copy"></i></a>
{ deleteButton }
</Dropdown.Menu>
</Dropdown>
}
}
let activeClass = ''
let isChecked = ''
if (checkedObjectsArray.indexOf(object.name) > -1) {
activeClass = ' fesl-row-selected'
isChecked = true
}
return (
<div key={ i } className={ "fesl-row " + loadingClass } data-type={ dataType(object.name, object.contentType) }>
<div className="fesl-item fi-name">
<div key={ i } className={ "fesl-row " + loadingClass + activeClass } data-type={ dataType(object.name, object.contentType) }>
<div className="fesl-item fesl-item-icon">
<div className="fi-select">
<input type="checkbox"
name={ object.name }
checked={ isChecked }
onChange={ (e) => checkObject(e, object.name) } />
<i className="fis-icon"></i>
<i className="fis-helper"></i>
</div>
</div>
<div className="fesl-item fesl-item-name">
<a href="" onClick={ (e) => selectPrefix(e, `${currentPath}${object.name}`) }>
{ object.name }
</a>
</div>
<div className="fesl-item fi-size">
<div className="fesl-item fesl-item-size">
{ size }
</div>
<div className="fesl-item fi-modified">
<div className="fesl-item fesl-item-modified">
{ lastModified }
</div>
<div className="fesl-item fi-actions">
<div className="fesl-item fesl-item-actions">
{ actionButtons }
</div>
</div>
@@ -72,4 +93,4 @@ export default connect(state => {
currentPath: state.currentPath,
loadPath: state.loadPath
}
})(ObjectsList)
})(ObjectsList)
+2 -2
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -29,7 +29,7 @@ let Path = ({currentBucket, currentPath, selectPrefix}) => {
}
return (
<h2><span className="main"><a onClick={ (e) => selectPrefix(e, '') } href="">{ currentBucket }</a></span>{ path }</h2>
<h2><span className="main"><a onClick={ (e) => selectPrefix(e, '') } href="">{ currentBucket }</a></span>{ path }</h2>
)
}
+1 -1
View File
@@ -77,4 +77,4 @@ class Policy extends Component {
}
}
export default connect(state => state)(Policy)
export default connect(state => state)(Policy)
+1 -1
View File
@@ -80,4 +80,4 @@ class PolicyInput extends Component {
}
}
export default connect(state => state)(PolicyInput)
export default connect(state => state)(PolicyInput)
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+4 -4
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -22,7 +22,7 @@ import connect from 'react-redux/lib/components/connect'
import logo from '../../img/logo.svg'
let SideBar = ({visibleBuckets, loadBucket, currentBucket, selectBucket, searchBuckets, landingPage, sidebarStatus, clickOutside, showPolicy}) => {
let SideBar = ({visibleBuckets, loadBucket, currentBucket, selectBucket, searchBuckets, sidebarStatus, clickOutside, showPolicy}) => {
const list = visibleBuckets.map((bucket, i) => {
return <li className={ classNames({
@@ -44,8 +44,8 @@ let SideBar = ({visibleBuckets, loadBucket, currentBucket, selectBucket, searchB
'toggled': sidebarStatus
}) }>
<div className="fes-header clearfix hidden-sm hidden-xs">
<a href="" onClick={ landingPage }><img src={ logo } alt="" />
<h2>Minio Browser</h2></a>
<img src={ logo } alt="" />
<h2>Minio Browser</h2>
</div>
<div className="fes-list">
<div className="input-group ig-dark ig-left ig-search" style={ { display: web.LoggedIn() ? 'block' : 'none' } }>
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+32 -4
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -21,6 +21,7 @@ export default (state = {
buckets: [],
visibleBuckets: [],
objects: [],
istruncated: true,
storageInfo: {},
serverInfo: {},
currentBucket: '',
@@ -53,9 +54,10 @@ export default (state = {
shareObject: {
show: false,
url: '',
expiry: 604800
object: ''
},
prefixWritable: false
prefixWritable: false,
checkedObjects: []
}, action) => {
let newState = Object.assign({}, state)
switch (action.type) {
@@ -76,7 +78,15 @@ export default (state = {
newState.currentBucket = action.currentBucket
break
case actions.SET_OBJECTS:
newState.objects = action.objects
if (!action.objects.length) {
newState.objects = []
newState.marker = ""
newState.istruncated = action.istruncated
} else {
newState.objects = [...action.objects]
newState.marker = action.marker
newState.istruncated = action.istruncated
}
break
case actions.SET_CURRENT_PATH:
newState.currentPath = action.currentPath
@@ -171,6 +181,24 @@ export default (state = {
case actions.SET_PREFIX_WRITABLE:
newState.prefixWritable = action.prefixWritable
break
case actions.REMOVE_OBJECT:
let idx = newState.objects.findIndex(object => object.name === action.object)
if (idx == -1) break
newState.objects = [...newState.objects.slice(0, idx), ...newState.objects.slice(idx + 1)]
break
case actions.CHECKED_OBJECTS_ADD:
newState.checkedObjects = [...newState.checkedObjects, action.objectName]
break
case actions.CHECKED_OBJECTS_REMOVE:
let index = newState.checkedObjects.indexOf(action.objectName)
if (index == -1) break
newState.checkedObjects = [...newState.checkedObjects.slice(0, index), ...newState.checkedObjects.slice(index + 1)]
break
case actions.CHECKED_OBJECTS_RESET:
newState.checkedObjects = []
break
}
return newState
}
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+4
View File
@@ -35,6 +35,10 @@
width: 100%;
}
.btn-white {
.btn-variant(#fff, darken(@text-color, 20%));
}
.btn-link {
.btn-variant(#eee, #545454);
}
+3 -5
View File
@@ -2,14 +2,13 @@
Header
----------------------------*/
.fe-header {
padding: 45px 55px 20px;
@media(min-width: @screen-md-min) {
@media(min-width: (@screen-sm-min - 100)) {
position: relative;
padding: 40px 40px 20px 45px;
}
@media(max-width: (@screen-xs-max - 100)) {
padding: 25px 25px 20px;
padding: 20px;
}
h2 {
@@ -239,4 +238,3 @@
}
+236 -102
View File
@@ -2,17 +2,19 @@
Row
----------------------------*/
.fesl-row {
padding-right: 40px;
padding-top: 5px;
padding-bottom: 5px;
position: relative;
@media (min-width: (@screen-sm-min - 100px)) {
padding: 5px 20px 5px 40px;
display: flex;
flex-flow: row nowrap;
justify-content: space-between;
}
@media(max-width: (@screen-xs-max - 100px)) {
padding: 5px 20px;
}
.clearfix();
}
@@ -20,7 +22,7 @@ header.fesl-row {
@media (min-width:(@screen-sm-min - 100px)) {
margin-bottom: 20px;
border-bottom: 1px solid lighten(@text-muted-color, 20%);
padding-left: 40px;
padding-left: 30px;
.fesl-item,
.fesli-sort {
@@ -42,7 +44,7 @@ header.fesl-row {
font-size: 14px;
}
&:hover:not(.fi-actions) {
&:hover:not(.fesl-item-actions) {
background: lighten(@text-muted-color, 22%);
color: @dark-gray;
@@ -58,54 +60,42 @@ header.fesl-row {
}
}
.list-type(@background, @icon) {
.fis-icon {
background-color: @background;
&:before {
content: @icon;
}
}
}
div.fesl-row {
padding-left: 85px;
border-bottom: 1px solid transparent;
cursor: default;
.transition(background-color);
.transition-duration(500ms);
@media (max-width: (@screen-xs-max - 100px)) {
padding-left: 70px;
padding-right: 45px;
padding: 5px 20px;
}
&:nth-child(even) {
background-color: #fafafa;
}
&:hover {
background-color: #fbf7dc;
}
&[data-type]:before {
font-family: @font-family-icon;
width: 35px;
height: 35px;
text-align: center;
line-height: 35px;
position: absolute;
border-radius: 50%;
font-size: 16px;
left: 50px;
top: 9px;
color: @white;
@media (max-width: (@screen-xs-max - 100px)) {
left: 20px;
&:not(.fesl-row-selected) {
&:nth-child(even) {
background-color: @list-row-even-bg;
}
}
&[data-type="folder"] {
@media (max-width: (@screen-xs-max - 100px)) {
.fesl-item {
&.fi-name {
padding-top: 10px;
padding-bottom: 7px;
}
&:hover {
.fis-icon {
&:before {
.opacity(0)
}
}
&.fi-size,
&.fi-modified {
display: none;
}
.fis-helper {
&:before {
.opacity(1);
}
}
}
@@ -113,54 +103,18 @@ div.fesl-row {
/*--------------------------
Icons
----------------------------*/
&[data-type=folder]:before {
content: '\f114';
background-color: #a1d6dd;
}
&[data-type=pdf]:before {
content: "\f1c1";
background-color: #fa7775;
}
&[data-type=zip]:before {
content: "\f1c6";
background-color: #427089;
}
&[data-type=audio]:before {
content: "\f1c7";
background-color: #009688
}
&[data-type=code]:before {
content: "\f1c9";
background-color: #997867;
}
&[data-type=excel]:before {
content: "\f1c3";
background-color: #64c866;
}
&[data-type=image]:before {
content: "\f1c5";
background-color: #f06292;
}
&[data-type=video]:before {
content: "\f1c8";
background-color: #f8c363;
}
&[data-type=other]:before {
content: "\f016";
background-color: #afafaf;
}
&[data-type=text]:before {
content: "\f0f6";
background-color: #8a8a8a;
}
&[data-type=doc]:before {
content: "\f1c2";
background-color: #2196f5;
}
&[data-type=presentation]:before {
content: "\f1c4";
background-color: #896ea6;
}
&[data-type=folder] { .list-type(#a1d6dd, '\f114'); }
&[data-type=pdf] {.list-type(#fa7775, '\f1c1'); }
&[data-type=zip] { .list-type(#427089, '\f1c6'); }
&[data-type=audio] { .list-type(#009688, '\f1c7'); }
&[data-type=code] { .list-type(#997867, "\f1c9"); }
&[data-type=excel] { .list-type(#f1c3, '\f1c3'); }
&[data-type=image] { .list-type(#f06292, '\f1c5'); }
&[data-type=video] { .list-type(#f8c363, '\f1c8'); }
&[data-type=other] { .list-type(#afafaf, '\f016'); }
&[data-type=text] { .list-type(#8a8a8a, '\f0f6'); }
&[data-type=doc] { .list-type(#2196f5, '\f1c2'); }
&[data-type=presentation] { .list-type(#896ea6, '\f1c4'); }
&.fesl-loading{
&:before {
@@ -180,6 +134,113 @@ div.fesl-row {
}
}
.fesl-row-selected {
background-color: @list-row-selected-bg;
&, .fesl-item a {
color: darken(@text-color, 10%);
}
}
.fi-select {
float: left;
position: relative;
width: 35px;
height: 35px;
margin: 3px 0;
@media(max-width: (@screen-xs-max - 100px)) {
margin-right: 15px;
}
input {
position: absolute;
left: 0;
top: 0;
width: 35px;
height: 35px;
z-index: 20;
opacity: 0;
cursor: pointer;
&:checked {
& ~ .fis-icon {
background-color: #32393F;
&:before {
opacity: 0;
}
}
& ~ .fis-helper {
&:before {
.scale(0);
}
&:after {
.scale(1);
}
}
}
}
}
.fis-icon {
display: inline-block;
vertical-align: top;
border-radius: 50%;
width: 35px;
height: 35px;
.transition(background-color);
.transition-duration(250ms);
&:before {
width: 100%;
height: 100%;
text-align: center;
position: absolute;
border-radius: 50%;
font-family: @font-family-icon;
line-height: 35px;
font-size: 16px;
color: @white;
.transition(all);
.transition-duration(300ms);
font-style: normal;
}
}
.fis-helper {
&:before,
&:after {
position: absolute;
.transition(all);
.transition-duration(250ms);
}
&:before {
content: '';
width: 15px;
height: 15px;
border: 2px solid @white;
z-index: 10;
border-radius: 2px;
top: 10px;
left: 10px;
opacity: 0;
}
&:after {
font-family: @font-family-icon;
content: '\f00c';
top: 8px;
left: 9px;
color: @white;
font-size: 14px;
.scale(0);
}
}
/*--------------------------
Files and Folders
@@ -192,26 +253,26 @@ div.fesl-row {
}
@media(min-width: (@screen-sm-min - 100px)) {
&:not(.fi-actions) {
&:not(.fesl-item-actions):not(.fesl-item-icon) {
text-overflow: ellipsis;
padding: 10px 15px;
white-space: nowrap;
overflow: hidden;
}
&.fi-name {
&.fesl-item-name {
flex: 3;
}
&.fi-size {
&.fesl-item-size {
width: 140px;
}
&.fi-modified {
&.fesl-item-modified {
width: 190px;
}
&.fi-actions {
&.fesl-item-actions {
width: 40px;
}
}
@@ -219,29 +280,29 @@ div.fesl-row {
@media(max-width: (@screen-xs-max - 100px)) {
padding: 0;
&.fi-name {
&.fesl-item-name {
width: 100%;
margin-bottom: 3px;
}
&.fi-size,
&.fi-modified {
&.fesl-item-size,
&.fesl-item-modified {
font-size: 12px;
color: #B5B5B5;
float: left;
}
&.fi-modified {
&.fesl-item-modified {
max-width: 72px;
white-space: nowrap;
overflow: hidden;
}
&.fi-size {
&.fesl-item-size {
margin-right: 10px;
}
&.fi-actions {
&.fesl-item-actions {
position: absolute;
top: 5px;
right: 10px;
@@ -266,7 +327,7 @@ div.fesl-row {
}
}
.fi-actions {
.fesl-item-actions {
.dropdown-menu {
background-color: transparent;
box-shadow: none;
@@ -324,6 +385,79 @@ div.fesl-row {
}
}
.list-actions {
position: fixed;
.translate3d(0, -100%, 0);
.opacity(0);
.transition(all);
.transition-duration(200ms);
padding: 20px 70px 20px 25px;
top: 0;
left: 0;
width: 100%;
background-color: @brand-primary;
z-index: 20;
box-shadow: 0 0 10px rgba(0,0,0,0.3);
text-align: center;
&.list-actions-toggled {
.translate3d(0, 0, 0);
.opacity(1);
}
}
.la-close {
position: absolute;
right: 20px;
top: 0;
color: #fff;
width: 30px;
height: 30px;
border-radius: 50%;
text-align: center;
line-height: 30px !important;
background: rgba(255, 255, 255, 0.1);
font-weight: normal;
bottom: 0;
margin: auto;
cursor: pointer;
&:hover {
background-color: rgba(255, 255, 255, 0.2);
}
}
.la-label {
color: @white;
float: left;
padding: 4px 0;
.fa {
font-size: 22px;
vertical-align: top;
margin-right: 10px;
margin-top: -1px;
}
}
.la-actions {
button {
background-color: transparent;
border: 2px solid rgba(255,255,255,0.9);
color: @white;
border-radius: 2px;
padding: 5px 10px;
font-size: 13px;
.transition(all);
.transition-duration(300ms);
margin-left: 10px;
&:hover {
background-color: @white;
color: @brand-primary;
}
}
}
@-webkit-keyframes fiad-action-anim {
from {
+14
View File
@@ -99,4 +99,18 @@
content: '7 days';
right: 0;
}
}
.modal-aheader {
height: 100px;
&:before {
height: 0 !important;
}
.modal-dialog {
margin: 0;
vertical-align: top;
}
}
+1 -1
View File
@@ -49,4 +49,4 @@
z-index: 1;
-webkit-animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
}
}
+8 -8
View File
@@ -7,7 +7,7 @@
position: fixed;
height: 100%;
overflow: hidden;
padding: 35px;
padding: 25px;
@media(min-width: @screen-md-min) {
.translate3d(0, 0, 0);
@@ -63,15 +63,15 @@
height: ~"calc(100vh - 260px)";
overflow: auto;
padding: 0;
margin: 0 -35px;
margin: 0 -25px;
& li {
position: relative;
& > a {
display: block;
padding: 10px 40px 12px 65px;
.text-overflow();
padding: 10px 45px 12px 55px;
word-wrap: break-word;
&:before {
font-family: FontAwesome;
@@ -79,7 +79,7 @@
font-size: 17px;
position: absolute;
top: 10px;
left: 35px;
left: 25px;
.opacity(0.8);
}
@@ -95,7 +95,7 @@
}
&.active {
background-color: rgba(0, 0, 0, 0.2);
background-color: #282e32;
& > a {
color: @white;
@@ -139,10 +139,10 @@
position: absolute;
top: 0;
right: 0;
width: 40px;
width: 35px;
height: 100%;
cursor: pointer;
background: url(../../img/more-h-light.svg) no-repeat left;
background: url(../../img/more-h-light.svg) no-repeat top 20px left;
}
/* Scrollbar */
+10 -3
View File
@@ -13,7 +13,7 @@
/*--------------------------
File Explorer
----------------------------*/
@fe-sidebar-width : 300px;
@fe-sidebar-width : 320px;
@text-muted-color : #BDBDBD;
@text-strong-color : #333;
@@ -81,7 +81,7 @@
/*-------------------------
Colors
--------------------------*/
@brand-primary: #2196F3;
@brand-primary: #2298f7;
@brand-success: #4CAF50;
@brand-info: #00BCD4;
@brand-warning: #FF9800;
@@ -91,4 +91,11 @@
/*-------------------------
Form
--------------------------*/
@input-border: #eee;
@input-border: #eee;
/*-------------------------
List
--------------------------*/
@list-row-selected-bg: #fbf2bf;
@list-row-even-bg: #fafafa;
+3 -3
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -70,9 +70,9 @@ async.waterfall([
commitId = stdout.replace('\n', '')
if (commitId.length !== 40) throw new Error('commitId invalid : ' + commitId)
assetsFileName = 'ui-assets.go';
var cmd = 'go-bindata-assetfs -pkg miniobrowser -nocompress=true production/...'
var cmd = 'go-bindata-assetfs -pkg browser -nocompress=true production/...'
if (!isProduction) {
cmd = 'go-bindata-assetfs -pkg miniobrowser -nocompress=true dev/...'
cmd = 'go-bindata-assetfs -pkg browser -nocompress=true dev/...'
}
console.log('Running', cmd)
exec(cmd, cb)
+7 -4
View File
@@ -1,5 +1,5 @@
{
"name": "minio-browser",
"name": "browser",
"version": "0.0.1",
"description": "Minio Browser",
"scripts": {
@@ -11,14 +11,14 @@
},
"repository": {
"type": "git",
"url": "https://github.com/minio/miniobrowser"
"url": "https://github.com/minio/minio"
},
"author": "Minio Inc",
"license": "Apache-2.0",
"bugs": {
"url": "https://github.com/minio/miniobrowser/issues"
"url": "https://github.com/minio/minio/issues"
},
"homepage": "https://github.com/minio/miniobrowser",
"homepage": "https://github.com/minio/minio",
"devDependencies": {
"async": "^1.5.2",
"babel-cli": "^6.14.0",
@@ -26,12 +26,14 @@
"babel-loader": "^6.2.5",
"babel-plugin-syntax-object-rest-spread": "^6.13.0",
"babel-plugin-transform-object-rest-spread": "^6.8.0",
"babel-polyfill": "^6.23.0",
"babel-preset-es2015": "^6.14.0",
"babel-preset-react": "^6.11.1",
"babel-register": "^6.14.0",
"copy-webpack-plugin": "^0.3.3",
"css-loader": "^0.23.1",
"esformatter": "^0.10.0",
"esformatter-jsx": "^7.4.1",
"esformatter-jsx-ignore": "^1.0.6",
"expect": "^1.20.2",
"history": "^1.17.0",
@@ -77,6 +79,7 @@
"react-custom-scrollbars": "^2.2.2",
"react-dom": "^0.14.6",
"react-dropzone": "^3.5.3",
"react-infinite-scroller": "^1.0.6",
"react-onclickout": "2.0.4"
}
}
+46 -46
View File
File diff suppressed because one or more lines are too long
+7 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -22,6 +22,7 @@ var purify = require("purifycss-webpack-plugin")
var exports = {
context: __dirname,
entry: [
"babel-polyfill",
path.resolve(__dirname, 'app/index.js')
],
output: {
@@ -71,6 +72,10 @@ var exports = {
target: 'http://localhost:9000',
secure: false
},
'/minio/zip': {
target: 'http://localhost:9000',
secure: false
}
}
},
plugins: [
@@ -96,6 +101,7 @@ var exports = {
if (process.env.NODE_ENV === 'dev') {
exports.entry = [
"babel-polyfill",
'webpack/hot/dev-server',
'webpack-dev-server/client?http://localhost:8080',
path.resolve(__dirname, 'app/index.js')
+64 -4
View File
@@ -694,6 +694,10 @@ babylon@^6.11.0, babylon@^6.15.0, babylon@^6.8.0:
version "6.15.0"
resolved "https://registry.yarnpkg.com/babylon/-/babylon-6.15.0.tgz#ba65cfa1a80e1759b0e89fb562e27dccae70348e"
babylon@6.14.1:
version "6.14.1"
resolved "https://registry.yarnpkg.com/babylon/-/babylon-6.14.1.tgz#956275fab72753ad9b3435d7afe58f8bf0a29815"
backo2@1.0.2:
version "1.0.2"
resolved "https://registry.yarnpkg.com/backo2/-/backo2-1.0.2.tgz#31ab1ac8b129363463e35b3ebb69f4dfcfba7947"
@@ -756,7 +760,7 @@ bluebird@^2.10.2, bluebird@^2.9.27:
version "2.11.0"
resolved "https://registry.yarnpkg.com/bluebird/-/bluebird-2.11.0.tgz#534b9033c022c9579c56ba3b3e5a5caafbb650e1"
bluebird@^3.4.7:
bluebird@^3.0.5, bluebird@^3.4.7:
version "3.4.7"
resolved "https://registry.yarnpkg.com/bluebird/-/bluebird-3.4.7.tgz#f72d760be09b7f76d08ed8fae98b289a8d05fab3"
@@ -1086,6 +1090,13 @@ concat-map@0.0.1:
version "0.0.1"
resolved "https://registry.yarnpkg.com/concat-map/-/concat-map-0.0.1.tgz#d8a96bd77fd68df7793a73036a3ba0d5405d477b"
config-chain@~1.1.5:
version "1.1.11"
resolved "https://registry.yarnpkg.com/config-chain/-/config-chain-1.1.11.tgz#aba09747dfbe4c3e70e766a6e41586e1859fc6f2"
dependencies:
ini "^1.3.4"
proto-list "~1.2.1"
connect-history-api-fallback@^1.3.0:
version "1.3.0"
resolved "https://registry.yarnpkg.com/connect-history-api-fallback/-/connect-history-api-fallback-1.3.0.tgz#e51d17f8f0ef0db90a64fdb47de3051556e9f169"
@@ -1462,6 +1473,15 @@ ecc-jsbn@~0.1.1:
dependencies:
jsbn "~0.1.0"
editorconfig@^0.13.2:
version "0.13.2"
resolved "https://registry.yarnpkg.com/editorconfig/-/editorconfig-0.13.2.tgz#8e57926d9ee69ab6cb999f027c2171467acceb35"
dependencies:
bluebird "^3.0.5"
commander "^2.9.0"
lru-cache "^3.2.0"
sigmund "^1.0.1"
ee-first@1.1.1:
version "1.1.1"
resolved "https://registry.yarnpkg.com/ee-first/-/ee-first-1.1.1.tgz#590c61156b0ae2f4f0255732a158b266bc56b21d"
@@ -1571,6 +1591,10 @@ escape-string-regexp@^1.0.2, escape-string-regexp@1.0.2:
version "1.0.2"
resolved "https://registry.yarnpkg.com/escape-string-regexp/-/escape-string-regexp-1.0.2.tgz#4dbc2fe674e71949caf3fb2695ce7f2dc1d9a8d1"
esformatter-ignore@^0.1.3:
version "0.1.3"
resolved "https://registry.yarnpkg.com/esformatter-ignore/-/esformatter-ignore-0.1.3.tgz#04d3b875bfa49dde004cc58df6f6bbc3c0567f1e"
esformatter-jsx-ignore@^1.0.6:
version "1.0.6"
resolved "https://registry.yarnpkg.com/esformatter-jsx-ignore/-/esformatter-jsx-ignore-1.0.6.tgz#e594f6b77db6f85d8c1179ae6dc465756d422489"
@@ -1578,6 +1602,15 @@ esformatter-jsx-ignore@^1.0.6:
esprima-fb "^12001.1.0-dev-harmony-fb"
fresh-falafel "^0.2.6"
esformatter-jsx@^7.4.1:
version "7.4.1"
resolved "https://registry.yarnpkg.com/esformatter-jsx/-/esformatter-jsx-7.4.1.tgz#b2209ae0908f413a747b1205727cbf4ba4249602"
dependencies:
babylon "6.14.1"
esformatter-ignore "^0.1.3"
extend "3.0.0"
js-beautify "1.6.4"
esformatter-parser@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/esformatter-parser/-/esformatter-parser-1.0.0.tgz#0854072d0487539ed39cae38d8a5432c17ec11d3"
@@ -2200,7 +2233,7 @@ inherits@2.0.1:
version "2.0.1"
resolved "https://registry.yarnpkg.com/inherits/-/inherits-2.0.1.tgz#b17d08d326b4423e568eff719f91b0b1cbdf69f1"
ini@~1.3.0:
ini@^1.3.4, ini@~1.3.0:
version "1.3.4"
resolved "https://registry.yarnpkg.com/ini/-/ini-1.3.4.tgz#0537cb79daf59b59a1a517dff706c86ec039162e"
@@ -2428,6 +2461,15 @@ js-base64@^2.1.9:
version "2.1.9"
resolved "https://registry.yarnpkg.com/js-base64/-/js-base64-2.1.9.tgz#f0e80ae039a4bd654b5f281fc93f04a914a7fcce"
js-beautify@1.6.4:
version "1.6.4"
resolved "https://registry.yarnpkg.com/js-beautify/-/js-beautify-1.6.4.tgz#a9af79699742ac9a1b6fddc1fdbc78bc4d515fc3"
dependencies:
config-chain "~1.1.5"
editorconfig "^0.13.2"
mkdirp "~0.5.0"
nopt "~3.0.1"
js-tokens@^3.0.0:
version "3.0.0"
resolved "https://registry.yarnpkg.com/js-tokens/-/js-tokens-3.0.0.tgz#a2f2a969caae142fb3cd56228358c89366957bd1"
@@ -2684,6 +2726,12 @@ lower-case@^1.1.1:
version "1.1.3"
resolved "https://registry.yarnpkg.com/lower-case/-/lower-case-1.1.3.tgz#c92393d976793eee5ba4edb583cf8eae35bd9bfb"
lru-cache@^3.2.0:
version "3.2.0"
resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-3.2.0.tgz#71789b3b7f5399bec8565dda38aa30d2a097efee"
dependencies:
pseudomap "^1.0.1"
lru-cache@2:
version "2.7.3"
resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-2.7.3.tgz#6d4524e8b955f95d4f5b58851ce21dd72fb4e952"
@@ -2902,7 +2950,7 @@ node-pre-gyp@^0.6.29:
tar "~2.2.1"
tar-pack "~3.3.0"
nopt@~3.0.6:
nopt@~3.0.1, nopt@~3.0.6:
version "3.0.6"
resolved "https://registry.yarnpkg.com/nopt/-/nopt-3.0.6.tgz#c6465dbf08abcd4db359317f79ac68a646b28ff9"
dependencies:
@@ -3447,6 +3495,10 @@ promise@^7.0.3, promise@^7.1.1:
dependencies:
asap "~2.0.3"
proto-list@~1.2.1:
version "1.2.4"
resolved "https://registry.yarnpkg.com/proto-list/-/proto-list-1.2.4.tgz#212d5bfe1318306a420f6402b8e26ff39647a849"
protochain@^1.0.5:
version "1.0.5"
resolved "https://registry.yarnpkg.com/protochain/-/protochain-1.0.5.tgz#991c407e99de264aadf8f81504b5e7faf7bfa260"
@@ -3462,6 +3514,10 @@ prr@~0.0.0:
version "0.0.0"
resolved "https://registry.yarnpkg.com/prr/-/prr-0.0.0.tgz#1a84b85908325501411853d0081ee3fa86e2926a"
pseudomap@^1.0.1:
version "1.0.2"
resolved "https://registry.yarnpkg.com/pseudomap/-/pseudomap-1.0.2.tgz#f052a28da70e618917ef0a8ac34c1ae5a68286b3"
punycode@^1.2.4, punycode@^1.4.1:
version "1.4.1"
resolved "https://registry.yarnpkg.com/punycode/-/punycode-1.4.1.tgz#c0d5a63b2718800ad8e1eb0fa5269c84dd41845e"
@@ -3609,6 +3665,10 @@ react-dropzone@^3.5.3:
dependencies:
attr-accept "^1.0.3"
react-infinite-scroller@^1.0.6:
version "1.0.6"
resolved "https://registry.yarnpkg.com/react-infinite-scroller/-/react-infinite-scroller-1.0.6.tgz#bb406d70032d09fa9e4a3e2175d3adbf4a3f559d"
react-onclickout@2.0.4:
version "2.0.4"
resolved "https://registry.yarnpkg.com/react-onclickout/-/react-onclickout-2.0.4.tgz#2c7539a647e1dcdcab0b28e2f4eae3c3e00f0c64"
@@ -4039,7 +4099,7 @@ shelljs@^0.7.0:
interpret "^1.0.0"
rechoir "^0.6.2"
sigmund@~1.0.0:
sigmund@^1.0.1, sigmund@~1.0.0:
version "1.0.1"
resolved "https://registry.yarnpkg.com/sigmund/-/sigmund-1.0.1.tgz#3ff21f198cad2175f9f3b781853fd94d0d19b590"
+3 -3
View File
@@ -53,7 +53,7 @@ go_build() {
release_bin_6="$release_str/$os-${arch}6vl/$(basename $package).$release_tag"
## Support building for ARM6vl
GOARM=6 GOOS=$os GOARCH=$arch go build --ldflags "${LDFLAGS}" -o $release_bin_6
GOARM=6 GOOS=$os GOARCH=$arch go build --ldflags "-s -w ${LDFLAGS}" -o $release_bin_6
## Copy
$CP -p $release_bin_6 $release_real_bin_6
@@ -70,7 +70,7 @@ go_build() {
release_bin_7="$release_str/$os-$arch/$(basename $package).$release_tag"
## Support building for ARM7vl
GOARM=7 GOOS=$os GOARCH=$arch go build --ldflags "${LDFLAGS}" -o $release_bin_7
GOARM=7 GOOS=$os GOARCH=$arch go build --ldflags "-s -w ${LDFLAGS}" -o $release_bin_7
## Copy
$CP -p $release_bin_7 $release_real_bin_7
@@ -82,7 +82,7 @@ go_build() {
shasum_str=$(${SHASUM} ${release_bin_7})
echo ${shasum_str} | $SED "s/$release_str\/$os-$arch\///g" > $release_shasum_7
else
GOOS=$os GOARCH=$arch go build --ldflags "${LDFLAGS}" -o $release_bin
GOOS=$os GOARCH=$arch go build --ldflags "-s -w ${LDFLAGS}" -o $release_bin
# Create copy
if [ $os == "windows" ]; then
+2 -2
View File
@@ -97,7 +97,7 @@ assert_is_supported_arch() {
assert_is_supported_os() {
case "${KNAME}" in
Linux | FreeBSD )
Linux | FreeBSD | OpenBSD | NetBSD | DragonFly )
return
;;
Darwin )
@@ -113,7 +113,7 @@ assert_is_supported_os() {
*)
echo "ERROR"
echo "OS '${KNAME}' is not supported."
echo "Supported OS: [Linux, FreeBSD, Darwin]"
echo "Supported OS: [Linux, FreeBSD, OpenBSD, NetBSD, Darwin, DragonFly]"
exit 1
esac
}
+351 -54
View File
@@ -17,6 +17,7 @@
package cmd
import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
@@ -28,22 +29,26 @@ import (
)
const (
minioAdminOpHeader = "X-Minio-Operation"
minioAdminOpHeader = "X-Minio-Operation"
minioConfigTmpFormat = "config-%s.json"
)
// Type-safe query params.
type mgmtQueryKey string
// Only valid query params for list/clear locks management APIs.
// Only valid query params for mgmt admin APIs.
const (
mgmtBucket mgmtQueryKey = "bucket"
mgmtObject mgmtQueryKey = "object"
mgmtPrefix mgmtQueryKey = "prefix"
mgmtOlderThan mgmtQueryKey = "older-than"
mgmtDelimiter mgmtQueryKey = "delimiter"
mgmtMarker mgmtQueryKey = "marker"
mgmtMaxKey mgmtQueryKey = "max-key"
mgmtDryRun mgmtQueryKey = "dry-run"
mgmtBucket mgmtQueryKey = "bucket"
mgmtObject mgmtQueryKey = "object"
mgmtPrefix mgmtQueryKey = "prefix"
mgmtLockDuration mgmtQueryKey = "duration"
mgmtDelimiter mgmtQueryKey = "delimiter"
mgmtMarker mgmtQueryKey = "marker"
mgmtKeyMarker mgmtQueryKey = "key-marker"
mgmtMaxKey mgmtQueryKey = "max-key"
mgmtDryRun mgmtQueryKey = "dry-run"
mgmtUploadIDMarker mgmtQueryKey = "upload-id-marker"
mgmtMaxUploads mgmtQueryKey = "max-uploads"
)
// ServerVersion - server version
@@ -54,8 +59,8 @@ type ServerVersion struct {
// ServerStatus - contains the response of service status API
type ServerStatus struct {
StorageInfo StorageInfo `json:"storageInfo"`
ServerVersion ServerVersion `json:"serverVersion"`
Uptime time.Duration `json:"uptime"`
}
// ServiceStatusHandler - GET /?service
@@ -70,15 +75,22 @@ func (adminAPI adminAPIHandlers) ServiceStatusHandler(w http.ResponseWriter, r *
return
}
// Fetch storage backend information
storageInfo := newObjectLayerFn().StorageInfo()
// Fetch server version
serverVersion := ServerVersion{Version: Version, CommitID: CommitID}
// Fetch uptimes from all peers. This may fail to due to lack
// of read-quorum availability.
uptime, err := getPeerUptimes(globalAdminPeers)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
errorIf(err, "Possibly failed to get uptime from majority of servers.")
return
}
// Create API response
serverStatus := ServerStatus{
StorageInfo: storageInfo,
ServerVersion: serverVersion,
Uptime: uptime,
}
// Marshal API response
@@ -131,8 +143,8 @@ func (adminAPI adminAPIHandlers) ServiceCredentialsHandler(w http.ResponseWriter
}
// Avoid setting new credentials when they are already passed
// by the environnement
if globalEnvAccessKey != "" || globalEnvSecretKey != "" {
// by the environment.
if globalIsEnvCreds {
writeErrorResponse(w, ErrMethodNotAllowed, r.URL)
return
}
@@ -153,25 +165,20 @@ func (adminAPI adminAPIHandlers) ServiceCredentialsHandler(w http.ResponseWriter
return
}
// Check passed credentials
cred, err := getCredential(req.Username, req.Password)
switch err {
case errInvalidAccessKeyLength:
writeErrorResponse(w, ErrAdminInvalidAccessKey, r.URL)
return
case errInvalidSecretKeyLength:
writeErrorResponse(w, ErrAdminInvalidSecretKey, r.URL)
creds, err := createCredential(req.Username, req.Password)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Notify all other Minio peers to update credentials
updateErrs := updateCredsOnPeers(cred)
updateErrs := updateCredsOnPeers(creds)
for peer, err := range updateErrs {
errorIf(err, "Unable to update credentials on peer %s.", peer)
}
// Update local credentials
serverConfig.SetCredential(cred)
// Update local credentials in memory.
serverConfig.SetCredential(creds)
if err = serverConfig.Save(); err != nil {
writeErrorResponse(w, ErrInternalError, r.URL)
return
@@ -181,11 +188,103 @@ func (adminAPI adminAPIHandlers) ServiceCredentialsHandler(w http.ResponseWriter
w.WriteHeader(http.StatusOK)
}
// ServerProperties holds some server information such as, version, region
// uptime, etc..
type ServerProperties struct {
Uptime time.Duration `json:"uptime"`
Version string `json:"version"`
CommitID string `json:"commitID"`
Region string `json:"region"`
SQSARN []string `json:"sqsARN"`
}
// ServerConnStats holds transferred bytes from/to the server
type ServerConnStats struct {
TotalInputBytes uint64 `json:"transferred"`
TotalOutputBytes uint64 `json:"received"`
Throughput uint64 `json:"throughput,omitempty"`
}
// ServerInfo holds the information that will be returned by ServerInfo API
type ServerInfo struct {
StorageInfo StorageInfo `json:"storage"`
ConnStats ServerConnStats `json:"network"`
Properties ServerProperties `json:"server"`
}
// ServerInfoHandler - GET /?server-info
// ----------
// Get server information
func (adminAPI adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *http.Request) {
// Authenticate request
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
storage := objLayer.StorageInfo()
// Build list of enabled ARNs queues
var arns []string
for queueArn := range globalEventNotifier.GetAllExternalTargets() {
arns = append(arns, queueArn)
}
// Fetch uptimes from all peers. This may fail to due to lack
// of read-quorum availability.
uptime, err := getPeerUptimes(globalAdminPeers)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
errorIf(err, "Unable to get uptime from majority of servers.")
return
}
// Build server properties information
properties := ServerProperties{
Version: Version,
CommitID: CommitID,
Region: serverConfig.GetRegion(),
SQSARN: arns,
Uptime: uptime,
}
// Build network info
connStats := ServerConnStats{
TotalInputBytes: globalConnStats.getTotalInputBytes(),
TotalOutputBytes: globalConnStats.getTotalOutputBytes(),
}
// Build the whole returned information
info := ServerInfo{
StorageInfo: storage,
ConnStats: connStats,
Properties: properties,
}
// Marshal API response
jsonBytes, err := json.Marshal(info)
if err != nil {
writeErrorResponse(w, ErrInternalError, r.URL)
errorIf(err, "Failed to marshal storage info into json.")
return
}
// Reply with storage information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
}
// validateLockQueryParams - Validates query params for list/clear locks management APIs.
func validateLockQueryParams(vars url.Values) (string, string, time.Duration, APIErrorCode) {
bucket := vars.Get(string(mgmtBucket))
prefix := vars.Get(string(mgmtPrefix))
relTimeStr := vars.Get(string(mgmtOlderThan))
durationStr := vars.Get(string(mgmtLockDuration))
// N B empty bucket name is invalid
if !IsValidBucketName(bucket) {
@@ -198,24 +297,24 @@ func validateLockQueryParams(vars url.Values) (string, string, time.Duration, AP
// If older-than parameter was empty then set it to 0s to list
// all locks older than now.
if relTimeStr == "" {
relTimeStr = "0s"
if durationStr == "" {
durationStr = "0s"
}
relTime, err := time.ParseDuration(relTimeStr)
duration, err := time.ParseDuration(durationStr)
if err != nil {
errorIf(err, "Failed to parse duration passed as query value.")
return "", "", time.Duration(0), ErrInvalidDuration
}
return bucket, prefix, relTime, ErrNone
return bucket, prefix, duration, ErrNone
}
// ListLocksHandler - GET /?lock&bucket=mybucket&prefix=myprefix&older-than=rel_time
// ListLocksHandler - GET /?lock&bucket=mybucket&prefix=myprefix&duration=duration
// - bucket is a mandatory query parameter
// - prefix and older-than are optional query parameters
// HTTP header x-minio-operation: list
// ---------
// Lists locks held on a given bucket, prefix and relative time.
// Lists locks held on a given bucket, prefix and duration it was held for.
func (adminAPI adminAPIHandlers) ListLocksHandler(w http.ResponseWriter, r *http.Request) {
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
@@ -224,15 +323,15 @@ func (adminAPI adminAPIHandlers) ListLocksHandler(w http.ResponseWriter, r *http
}
vars := r.URL.Query()
bucket, prefix, relTime, adminAPIErr := validateLockQueryParams(vars)
bucket, prefix, duration, adminAPIErr := validateLockQueryParams(vars)
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// Fetch lock information of locks matching bucket/prefix that
// are available since relTime.
volLocks, err := listPeerLocksInfo(globalAdminPeers, bucket, prefix, relTime)
// are available for longer than duration.
volLocks, err := listPeerLocksInfo(globalAdminPeers, bucket, prefix, duration)
if err != nil {
writeErrorResponse(w, ErrInternalError, r.URL)
errorIf(err, "Failed to fetch lock information from remote nodes.")
@@ -248,16 +347,16 @@ func (adminAPI adminAPIHandlers) ListLocksHandler(w http.ResponseWriter, r *http
}
// Reply with list of locks held on bucket, matching prefix
// older than relTime supplied, as json.
// held longer than duration supplied, as json.
writeSuccessResponseJSON(w, jsonBytes)
}
// ClearLocksHandler - POST /?lock&bucket=mybucket&prefix=myprefix&older-than=relTime
// ClearLocksHandler - POST /?lock&bucket=mybucket&prefix=myprefix&duration=duration
// - bucket is a mandatory query parameter
// - prefix and older-than are optional query parameters
// HTTP header x-minio-operation: clear
// ---------
// Clear locks held on a given bucket, prefix and relative time.
// Clear locks held on a given bucket, prefix and duration it was held for.
func (adminAPI adminAPIHandlers) ClearLocksHandler(w http.ResponseWriter, r *http.Request) {
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
@@ -266,15 +365,15 @@ func (adminAPI adminAPIHandlers) ClearLocksHandler(w http.ResponseWriter, r *htt
}
vars := r.URL.Query()
bucket, prefix, relTime, adminAPIErr := validateLockQueryParams(vars)
bucket, prefix, duration, adminAPIErr := validateLockQueryParams(vars)
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// Fetch lock information of locks matching bucket/prefix that
// are available since relTime.
volLocks, err := listPeerLocksInfo(globalAdminPeers, bucket, prefix, relTime)
// are held for longer than duration.
volLocks, err := listPeerLocksInfo(globalAdminPeers, bucket, prefix, duration)
if err != nil {
writeErrorResponse(w, ErrInternalError, r.URL)
errorIf(err, "Failed to fetch lock information from remote nodes.")
@@ -289,7 +388,7 @@ func (adminAPI adminAPIHandlers) ClearLocksHandler(w http.ResponseWriter, r *htt
return
}
// Remove lock matching bucket/prefix older than relTime.
// Remove lock matching bucket/prefix held longer than duration.
for _, volLock := range volLocks {
globalNSMutex.ForceUnlock(volLock.Bucket, volLock.Object)
}
@@ -298,8 +397,57 @@ func (adminAPI adminAPIHandlers) ClearLocksHandler(w http.ResponseWriter, r *htt
writeSuccessResponseJSON(w, jsonBytes)
}
// validateHealQueryParams - Validates query params for heal list management API.
func validateHealQueryParams(vars url.Values) (string, string, string, string, int, APIErrorCode) {
// ListUploadsHealHandler - similar to listObjectsHealHandler
// GET
// /?heal&bucket=mybucket&prefix=myprefix&key-marker=mymarker&upload-id-marker=myuploadid&delimiter=mydelimiter&max-uploads=1000
// - bucket is mandatory query parameter
// - rest are optional query parameters List upto maxKey objects that
// need healing in a given bucket matching the given prefix.
func (adminAPI adminAPIHandlers) ListUploadsHealHandler(w http.ResponseWriter, r *http.Request) {
// Get object layer instance.
objLayer := newObjectLayerFn()
if objLayer == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
// Validate request signature.
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// Validate query params.
vars := r.URL.Query()
bucket := vars.Get(string(mgmtBucket))
prefix, keyMarker, uploadIDMarker, delimiter, maxUploads, _ := getBucketMultipartResources(r.URL.Query())
if err := checkListMultipartArgs(bucket, prefix, keyMarker, uploadIDMarker, delimiter, objLayer); err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
if maxUploads <= 0 || maxUploads > maxUploadsList {
writeErrorResponse(w, ErrInvalidMaxUploads, r.URL)
return
}
// Get the list objects to be healed.
listMultipartInfos, err := objLayer.ListUploadsHeal(bucket, prefix,
keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
listResponse := generateListMultipartUploadsResponse(bucket, listMultipartInfos)
// Write success response.
writeSuccessResponseXML(w, encodeResponse(listResponse))
}
// extractListObjectsHealQuery - Validates query params for heal objects list management API.
func extractListObjectsHealQuery(vars url.Values) (string, string, string, string, int, APIErrorCode) {
bucket := vars.Get(string(mgmtBucket))
prefix := vars.Get(string(mgmtPrefix))
marker := vars.Get(string(mgmtMarker))
@@ -316,10 +464,13 @@ func validateHealQueryParams(vars url.Values) (string, string, string, string, i
return "", "", "", "", 0, ErrInvalidObjectName
}
// check if maxKey is a valid integer.
maxKey, err := strconv.Atoi(maxKeyStr)
if err != nil {
return "", "", "", "", 0, ErrInvalidMaxKeys
// check if maxKey is a valid integer, if present.
var maxKey int
var err error
if maxKeyStr != "" {
if maxKey, err = strconv.Atoi(maxKeyStr); err != nil {
return "", "", "", "", 0, ErrInvalidMaxKeys
}
}
// Validate prefix, marker, delimiter and maxKey.
@@ -352,7 +503,7 @@ func (adminAPI adminAPIHandlers) ListObjectsHealHandler(w http.ResponseWriter, r
// Validate query params.
vars := r.URL.Query()
bucket, prefix, marker, delimiter, maxKey, adminAPIErr := validateHealQueryParams(vars)
bucket, prefix, marker, delimiter, maxKey, adminAPIErr := extractListObjectsHealQuery(vars)
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
@@ -541,7 +692,6 @@ func (adminAPI adminAPIHandlers) HealFormatHandler(w http.ResponseWriter, r *htt
// Create a new set of storage instances to heal format.json.
bootstrapDisks, err := initStorageDisks(globalEndpoints)
if err != nil {
fmt.Println(traceError(err))
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -549,7 +699,6 @@ func (adminAPI adminAPIHandlers) HealFormatHandler(w http.ResponseWriter, r *htt
// Heal format.json on available storage.
err = healFormatXL(bootstrapDisks)
if err != nil {
fmt.Println(traceError(err))
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -557,7 +706,6 @@ func (adminAPI adminAPIHandlers) HealFormatHandler(w http.ResponseWriter, r *htt
// Instantiate new object layer with newly formatted storage.
newObjectAPI, err := newXLObjects(bootstrapDisks)
if err != nil {
fmt.Println(traceError(err))
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -576,3 +724,152 @@ func (adminAPI adminAPIHandlers) HealFormatHandler(w http.ResponseWriter, r *htt
// Return 200 on success.
writeSuccessResponseHeadersOnly(w)
}
// GetConfigHandler - GET /?config
// - x-minio-operation = get
// Get config.json of this minio setup.
func (adminAPI adminAPIHandlers) GetConfigHandler(w http.ResponseWriter, r *http.Request) {
// Validate request signature.
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// check if objectLayer is initialized, if not return.
if newObjectLayerFn() == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
// Get config.json from all nodes. In a single node setup, it
// returns local config.json.
configBytes, err := getPeerConfig(globalAdminPeers)
if err != nil {
errorIf(err, "Failed to get config from peers")
writeErrorResponse(w, toAdminAPIErrCode(err), r.URL)
return
}
writeSuccessResponseJSON(w, configBytes)
}
// toAdminAPIErrCode - converts errXLWriteQuorum error to admin API
// specific error.
func toAdminAPIErrCode(err error) APIErrorCode {
switch err {
case errXLWriteQuorum:
return ErrAdminConfigNoQuorum
}
return toAPIErrorCode(err)
}
// SetConfigResult - represents detailed results of a set-config
// operation.
type nodeSummary struct {
Name string `json:"name"`
ErrSet bool `json:"errSet"`
ErrMsg string `json:"errMsg"`
}
type setConfigResult struct {
NodeResults []nodeSummary `json:"nodeResults"`
Status bool `json:"status"`
}
// writeSetConfigResponse - writes setConfigResult value as json depending on the status.
func writeSetConfigResponse(w http.ResponseWriter, peers adminPeers, errs []error, status bool, reqURL *url.URL) {
var nodeResults []nodeSummary
// Build nodeResults based on error values received during
// set-config operation.
for i := range errs {
nodeResults = append(nodeResults, nodeSummary{
Name: peers[i].addr,
ErrSet: errs[i] != nil,
ErrMsg: fmt.Sprintf("%v", errs[i]),
})
}
result := setConfigResult{
Status: status,
NodeResults: nodeResults,
}
// The following elaborate json encoding is to avoid escaping
// '<', '>' in <nil>. Note: json.Encoder.Encode() adds a
// gratuitous "\n".
var resultBuf bytes.Buffer
enc := json.NewEncoder(&resultBuf)
enc.SetEscapeHTML(false)
jsonErr := enc.Encode(result)
if jsonErr != nil {
writeErrorResponse(w, toAPIErrorCode(jsonErr), reqURL)
return
}
writeSuccessResponseJSON(w, resultBuf.Bytes())
return
}
// SetConfigHandler - PUT /?config
// - x-minio-operation = set
func (adminAPI adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Request) {
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
// Validate request signature.
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// Read configuration bytes from request body.
configBytes, err := ioutil.ReadAll(r.Body)
if err != nil {
errorIf(err, "Failed to read config from request body.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Write config received from request onto a temporary file on
// all nodes.
tmpFileName := fmt.Sprintf(minioConfigTmpFormat, mustGetUUID())
errs := writeTmpConfigPeers(globalAdminPeers, tmpFileName, configBytes)
// Check if the operation succeeded in quorum or more nodes.
rErr := reduceWriteQuorumErrs(errs, nil, len(globalAdminPeers)/2+1)
if rErr != nil {
writeSetConfigResponse(w, globalAdminPeers, errs, false, r.URL)
return
}
// Take a lock on minio/config.json. NB minio is a reserved
// bucket name and wouldn't conflict with normal object
// operations.
configLock := globalNSMutex.NewNSLock(minioReservedBucket, minioConfigFile)
configLock.Lock()
defer configLock.Unlock()
// Rename the temporary config file to config.json
errs = commitConfigPeers(globalAdminPeers, tmpFileName)
rErr = reduceWriteQuorumErrs(errs, nil, len(globalAdminPeers)/2+1)
if rErr != nil {
writeSetConfigResponse(w, globalAdminPeers, errs, false, r.URL)
return
}
// serverMux (cmd/server-mux.go) implements graceful shutdown,
// where all listeners are closed and process restart/shutdown
// happens after 5s or completion of all ongoing http
// requests, whichever is earlier.
writeSetConfigResponse(w, globalAdminPeers, errs, true, r.URL)
// Restart all node for the modified config to take effect.
sendServiceCmd(globalAdminPeers, serviceRestart)
}
+463 -28
View File
@@ -20,15 +20,113 @@ import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
"io/ioutil"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"time"
router "github.com/gorilla/mux"
)
var configJSON = []byte(`{
"version": "13",
"credential": {
"accessKey": "minio",
"secretKey": "minio123"
},
"region": "us-west-1",
"logger": {
"console": {
"enable": true,
"level": "fatal"
},
"file": {
"enable": false,
"fileName": "",
"level": ""
}
},
"notify": {
"amqp": {
"1": {
"enable": false,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"streaming": {
"enable": false,
"clusterID": "",
"clientID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"elasticsearch": {
"1": {
"enable": false,
"url": "",
"index": ""
}
},
"redis": {
"1": {
"enable": false,
"address": "",
"password": "",
"key": ""
}
},
"postgresql": {
"1": {
"enable": false,
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": ""
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
}
}
}
}`)
// adminXLTestBed - encapsulates subsystems that need to be setup for
// admin-handler unit tests.
type adminXLTestBed struct {
@@ -43,6 +141,7 @@ type adminXLTestBed struct {
func prepareAdminXLTestBed() (*adminXLTestBed, error) {
// reset global variables to start afresh.
resetTestGlobals()
// Initialize minio server config.
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
@@ -54,6 +153,9 @@ func prepareAdminXLTestBed() (*adminXLTestBed, error) {
return nil, xlErr
}
// Initialize boot time
globalBootTime = time.Now().UTC()
// Set globalEndpoints for a single node XL setup.
for _, xlDir := range xlDirs {
globalEndpoints = append(globalEndpoints, &url.URL{
@@ -197,7 +299,7 @@ func testServicesCmdHandler(cmd cmdType, t *testing.T) {
// Initialize admin peers to make admin RPC calls. Note: In a
// single node setup, this degenerates to a simple function
// call under the hood.
eps, err := parseStorageEndpoints([]string{"http://localhost"})
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
@@ -224,14 +326,13 @@ func testServicesCmdHandler(cmd cmdType, t *testing.T) {
if cmd == statusCmd {
expectedInfo := ServerStatus{
StorageInfo: newObjectLayerFn().StorageInfo(),
ServerVersion: ServerVersion{Version: Version, CommitID: CommitID},
}
receivedInfo := ServerStatus{}
if jsonErr := json.Unmarshal(rec.Body.Bytes(), &receivedInfo); jsonErr != nil {
t.Errorf("Failed to unmarshal StorageInfo - %v", jsonErr)
}
if expectedInfo != receivedInfo {
if expectedInfo.ServerVersion != receivedInfo.ServerVersion {
t.Errorf("Expected storage info and received storage info differ, %v %v", expectedInfo, receivedInfo)
}
}
@@ -264,7 +365,7 @@ func TestServiceSetCreds(t *testing.T) {
// Initialize admin peers to make admin RPC calls. Note: In a
// single node setup, this degenerates to a simple function
// call under the hood.
eps, err := parseStorageEndpoints([]string{"http://localhost"})
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
@@ -294,11 +395,9 @@ func TestServiceSetCreds(t *testing.T) {
for i, testCase := range testCases {
// Set or unset environement keys
if !testCase.EnvKeysSet {
globalEnvAccessKey = ""
globalEnvSecretKey = ""
globalIsEnvCreds = false
} else {
globalEnvAccessKey = testCase.Username
globalEnvSecretKey = testCase.Password
globalIsEnvCreds = true
}
// Construct setCreds request body
@@ -336,12 +435,12 @@ func TestServiceSetCreds(t *testing.T) {
}
// mkLockQueryVal - helper function to build lock query param.
func mkLockQueryVal(bucket, prefix, relTimeStr string) url.Values {
func mkLockQueryVal(bucket, prefix, durationStr string) url.Values {
qVal := url.Values{}
qVal.Set("lock", "")
qVal.Set(string(mgmtBucket), bucket)
qVal.Set(string(mgmtPrefix), prefix)
qVal.Set(string(mgmtOlderThan), relTimeStr)
qVal.Set(string(mgmtLockDuration), durationStr)
return qVal
}
@@ -354,7 +453,7 @@ func TestListLocksHandler(t *testing.T) {
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
eps, err := parseStorageEndpoints([]string{"http://localhost"})
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
@@ -366,41 +465,41 @@ func TestListLocksHandler(t *testing.T) {
testCases := []struct {
bucket string
prefix string
relTime string
duration string
expectedStatus int
}{
// Test 1 - valid testcase
{
bucket: "mybucket",
prefix: "myobject",
relTime: "1s",
duration: "1s",
expectedStatus: http.StatusOK,
},
// Test 2 - invalid duration
{
bucket: "mybucket",
prefix: "myprefix",
relTime: "invalidDuration",
duration: "invalidDuration",
expectedStatus: http.StatusBadRequest,
},
// Test 3 - invalid bucket name
{
bucket: `invalid\\Bucket`,
prefix: "myprefix",
relTime: "1h",
duration: "1h",
expectedStatus: http.StatusBadRequest,
},
// Test 4 - invalid prefix
{
bucket: "mybucket",
prefix: `invalid\\Prefix`,
relTime: "1h",
duration: "1h",
expectedStatus: http.StatusBadRequest,
},
}
for i, test := range testCases {
queryVal := mkLockQueryVal(test.bucket, test.prefix, test.relTime)
queryVal := mkLockQueryVal(test.bucket, test.prefix, test.duration)
req, err := newTestRequest("GET", "/?"+queryVal.Encode(), 0, nil)
if err != nil {
t.Fatalf("Test %d - Failed to construct list locks request - %v", i+1, err)
@@ -429,7 +528,7 @@ func TestClearLocksHandler(t *testing.T) {
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
eps, err := parseStorageEndpoints([]string{"http://localhost"})
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
@@ -438,41 +537,41 @@ func TestClearLocksHandler(t *testing.T) {
testCases := []struct {
bucket string
prefix string
relTime string
duration string
expectedStatus int
}{
// Test 1 - valid testcase
{
bucket: "mybucket",
prefix: "myobject",
relTime: "1s",
duration: "1s",
expectedStatus: http.StatusOK,
},
// Test 2 - invalid duration
{
bucket: "mybucket",
prefix: "myprefix",
relTime: "invalidDuration",
duration: "invalidDuration",
expectedStatus: http.StatusBadRequest,
},
// Test 3 - invalid bucket name
{
bucket: `invalid\\Bucket`,
prefix: "myprefix",
relTime: "1h",
duration: "1h",
expectedStatus: http.StatusBadRequest,
},
// Test 4 - invalid prefix
{
bucket: "mybucket",
prefix: `invalid\\Prefix`,
relTime: "1h",
duration: "1h",
expectedStatus: http.StatusBadRequest,
},
}
for i, test := range testCases {
queryVal := mkLockQueryVal(test.bucket, test.prefix, test.relTime)
queryVal := mkLockQueryVal(test.bucket, test.prefix, test.duration)
req, err := newTestRequest("POST", "/?"+queryVal.Encode(), 0, nil)
if err != nil {
t.Fatalf("Test %d - Failed to construct clear locks request - %v", i+1, err)
@@ -641,7 +740,7 @@ func TestValidateHealQueryParams(t *testing.T) {
}
for i, test := range testCases {
vars := mkListObjectsQueryVal(test.bucket, test.prefix, test.marker, test.delimiter, test.maxKeys)
_, _, _, _, _, actualErr := validateHealQueryParams(vars)
_, _, _, _, _, actualErr := extractListObjectsHealQuery(vars)
if actualErr != test.apiErr {
t.Errorf("Test %d - Expected %v but received %v",
i+1, getAPIError(test.apiErr), getAPIError(actualErr))
@@ -757,9 +856,6 @@ func TestListObjectsHealHandler(t *testing.T) {
}
for i, test := range testCases {
if i != 0 {
continue
}
queryVal := mkListObjectsQueryVal(test.bucket, test.prefix, test.marker, test.delimiter, test.maxKeys)
req, err := newTestRequest("GET", "/?"+queryVal.Encode(), 0, nil)
if err != nil {
@@ -986,3 +1082,342 @@ func TestHealFormatHandler(t *testing.T) {
t.Errorf("Expected to succeed but failed with %d", rec.Code)
}
}
// TestGetConfigHandler - test for GetConfigHandler.
func TestGetConfigHandler(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
// Set globalMinioAddr to be able to distinguish local endpoints from remote.
globalMinioAddr = eps[0].Host
initGlobalAdminPeers(eps)
// Prepare query params for get-config mgmt REST API.
queryVal := url.Values{}
queryVal.Set("config", "")
req, err := newTestRequest("GET", "/?"+queryVal.Encode(), 0, nil)
if err != nil {
t.Fatalf("Failed to construct get-config object request - %v", err)
}
// Set x-minio-operation header to get.
req.Header.Set(minioAdminOpHeader, "get")
// Sign the request using signature v4.
cred := serverConfig.GetCredential()
err = signRequestV4(req, cred.AccessKey, cred.SecretKey)
if err != nil {
t.Fatalf("Failed to sign heal object request - %v", err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("Expected to succeed but failed with %d", rec.Code)
}
}
// TestSetConfigHandler - test for SetConfigHandler.
func TestSetConfigHandler(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
// Set globalMinioAddr to be able to distinguish local endpoints from remote.
globalMinioAddr = eps[0].Host
initGlobalAdminPeers(eps)
// SetConfigHandler restarts minio setup - need to start a
// signal receiver to receive on globalServiceSignalCh.
go testServiceSignalReceiver(restartCmd, t)
// Prepare query params for set-config mgmt REST API.
queryVal := url.Values{}
queryVal.Set("config", "")
req, err := newTestRequest("PUT", "/?"+queryVal.Encode(), int64(len(configJSON)), bytes.NewReader(configJSON))
if err != nil {
t.Fatalf("Failed to construct get-config object request - %v", err)
}
// Set x-minio-operation header to set.
req.Header.Set(minioAdminOpHeader, "set")
// Sign the request using signature v4.
cred := serverConfig.GetCredential()
err = signRequestV4(req, cred.AccessKey, cred.SecretKey)
if err != nil {
t.Fatalf("Failed to sign heal object request - %v", err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("Expected to succeed but failed with %d", rec.Code)
}
result := setConfigResult{}
err = json.NewDecoder(rec.Body).Decode(&result)
if err != nil {
t.Fatalf("Failed to decode set config result json %v", err)
}
if !result.Status {
t.Error("Expected set-config to succeed, but failed")
}
}
// TestToAdminAPIErr - test for toAdminAPIErr helper function.
func TestToAdminAPIErr(t *testing.T) {
testCases := []struct {
err error
expectedAPIErr APIErrorCode
}{
// 1. Server not in quorum.
{
err: errXLWriteQuorum,
expectedAPIErr: ErrAdminConfigNoQuorum,
},
// 2. No error.
{
err: nil,
expectedAPIErr: ErrNone,
},
// 3. Non-admin API specific error.
{
err: errDiskNotFound,
expectedAPIErr: toAPIErrorCode(errDiskNotFound),
},
}
for i, test := range testCases {
actualErr := toAdminAPIErrCode(test.err)
if actualErr != test.expectedAPIErr {
t.Errorf("Test %d: Expected %v but received %v",
i+1, test.expectedAPIErr, actualErr)
}
}
}
func TestWriteSetConfigResponse(t *testing.T) {
testCases := []struct {
status bool
errs []error
}{
// 1. all nodes returned success.
{
status: true,
errs: []error{nil, nil, nil, nil},
},
// 2. some nodes returned errors.
{
status: false,
errs: []error{errDiskNotFound, nil, errDiskAccessDenied, errFaultyDisk},
},
}
testPeers := []adminPeer{
{
addr: "localhost:9001",
},
{
addr: "localhost:9002",
},
{
addr: "localhost:9003",
},
{
addr: "localhost:9004",
},
}
testURL, err := url.Parse("dummy.com")
if err != nil {
t.Fatalf("Failed to parse a place-holder url")
}
var actualResult setConfigResult
for i, test := range testCases {
rec := httptest.NewRecorder()
writeSetConfigResponse(rec, testPeers, test.errs, test.status, testURL)
resp := rec.Result()
jsonBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("Test %d: Failed to read response %v", i+1, err)
}
err = json.Unmarshal(jsonBytes, &actualResult)
if err != nil {
t.Fatalf("Test %d: Failed to unmarshal json %v", i+1, err)
}
if actualResult.Status != test.status {
t.Errorf("Test %d: Expected status %v but received %v", i+1, test.status, actualResult.Status)
}
for p, res := range actualResult.NodeResults {
if res.Name != testPeers[p].addr {
t.Errorf("Test %d: Expected node name %s but received %s", i+1, testPeers[p].addr, res.Name)
}
expectedErrMsg := fmt.Sprintf("%v", test.errs[p])
if res.ErrMsg != expectedErrMsg {
t.Errorf("Test %d: Expected error %s but received %s", i+1, expectedErrMsg, res.ErrMsg)
}
expectedErrSet := test.errs[p] != nil
if res.ErrSet != expectedErrSet {
t.Errorf("Test %d: Expected ErrSet %v but received %v", i+1, expectedErrSet, res.ErrSet)
}
}
}
}
// mkUploadsHealQuery - helper function to construct query values for
// listUploadsHeal.
func mkUploadsHealQuery(bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploadsStr string) url.Values {
queryVal := make(url.Values)
queryVal.Set("heal", "")
queryVal.Set(string(mgmtBucket), bucket)
queryVal.Set(string(mgmtPrefix), prefix)
queryVal.Set(string(mgmtKeyMarker), keyMarker)
queryVal.Set(string(mgmtUploadIDMarker), uploadIDMarker)
queryVal.Set(string(mgmtDelimiter), delimiter)
queryVal.Set(string(mgmtMaxUploads), maxUploadsStr)
return queryVal
}
func TestListHealUploadsHandler(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
err = adminTestBed.objLayer.MakeBucket("mybucket")
if err != nil {
t.Fatalf("Failed to make bucket - %v", err)
}
// Delete bucket after running all test cases.
defer adminTestBed.objLayer.DeleteBucket("mybucket")
testCases := []struct {
bucket string
prefix string
keyMarker string
delimiter string
maxKeys string
statusCode int
}{
// 1. Valid params.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "10",
statusCode: http.StatusOK,
},
// 2. Valid params with empty prefix.
{
bucket: "mybucket",
prefix: "",
keyMarker: "",
delimiter: "/",
maxKeys: "10",
statusCode: http.StatusOK,
},
// 3. Invalid params with invalid bucket.
{
bucket: `invalid\\Bucket`,
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "10",
statusCode: getAPIError(ErrInvalidBucketName).HTTPStatusCode,
},
// 4. Invalid params with invalid prefix.
{
bucket: "mybucket",
prefix: `invalid\\Prefix`,
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "10",
statusCode: getAPIError(ErrInvalidObjectName).HTTPStatusCode,
},
// 5. Invalid params with invalid maxKeys.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "-1",
statusCode: getAPIError(ErrInvalidMaxUploads).HTTPStatusCode,
},
// 6. Invalid params with unsupported prefix marker combination.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "notmatchingmarker",
delimiter: "/",
maxKeys: "10",
statusCode: getAPIError(ErrNotImplemented).HTTPStatusCode,
},
// 7. Invalid params with unsupported delimiter.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "notmatchingmarker",
delimiter: "unsupported",
maxKeys: "10",
statusCode: getAPIError(ErrNotImplemented).HTTPStatusCode,
},
// 8. Invalid params with invalid max Keys
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "999999999999999999999999999",
statusCode: getAPIError(ErrInvalidMaxUploads).HTTPStatusCode,
},
}
for i, test := range testCases {
queryVal := mkUploadsHealQuery(test.bucket, test.prefix, test.keyMarker, "", test.delimiter, test.maxKeys)
req, err := newTestRequest("GET", "/?"+queryVal.Encode(), 0, nil)
if err != nil {
t.Fatalf("Test %d - Failed to construct list uploads needing heal request - %v", i+1, err)
}
req.Header.Set(minioAdminOpHeader, "list-uploads")
cred := serverConfig.GetCredential()
err = signRequestV4(req, cred.AccessKey, cred.SecretKey)
if err != nil {
t.Fatalf("Test %d - Failed to sign list uploads needing heal request - %v", i+1, err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if test.statusCode != rec.Code {
t.Errorf("Test %d - Expected HTTP status code %d but received %d", i+1, test.statusCode, rec.Code)
}
}
}
+12
View File
@@ -39,6 +39,9 @@ func registerAdminRouter(mux *router.Router) {
// Service update credentials
adminRouter.Methods("POST").Queries("service", "").Headers(minioAdminOpHeader, "set-credentials").HandlerFunc(adminAPI.ServiceCredentialsHandler)
// Info operations
adminRouter.Methods("GET").Queries("info", "").HandlerFunc(adminAPI.ServerInfoHandler)
/// Lock operations
// List Locks
@@ -50,6 +53,8 @@ func registerAdminRouter(mux *router.Router) {
// List Objects needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-objects").HandlerFunc(adminAPI.ListObjectsHealHandler)
// List Uploads needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-uploads").HandlerFunc(adminAPI.ListUploadsHealHandler)
// List Buckets needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-buckets").HandlerFunc(adminAPI.ListBucketsHealHandler)
@@ -59,4 +64,11 @@ func registerAdminRouter(mux *router.Router) {
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "object").HandlerFunc(adminAPI.HealObjectHandler)
// Heal Format.
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "format").HandlerFunc(adminAPI.HealFormatHandler)
/// Config operations
// Get config
adminRouter.Methods("GET").Queries("config", "").Headers(minioAdminOpHeader, "get").HandlerFunc(adminAPI.GetConfigHandler)
// Set Config
adminRouter.Methods("PUT").Queries("config", "").Headers(minioAdminOpHeader, "set").HandlerFunc(adminAPI.SetConfigHandler)
}
+381 -15
View File
@@ -17,12 +17,30 @@
package cmd
import (
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
"path"
"path/filepath"
"reflect"
"sort"
"sync"
"time"
)
const (
// Admin service names
serviceRestartRPC = "Admin.Restart"
listLocksRPC = "Admin.ListLocks"
reInitDisksRPC = "Admin.ReInitDisks"
uptimeRPC = "Admin.Uptime"
getConfigRPC = "Admin.GetConfig"
writeTmpConfigRPC = "Admin.WriteTmpConfig"
commitConfigRPC = "Admin.CommitConfig"
)
// localAdminClient - represents admin operation to be executed locally.
type localAdminClient struct {
}
@@ -37,8 +55,12 @@ type remoteAdminClient struct {
// commands like service stop and service restart.
type adminCmdRunner interface {
Restart() error
ListLocks(bucket, prefix string, relTime time.Duration) ([]VolumeLockInfo, error)
ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error)
ReInitDisks() error
Uptime() (time.Duration, error)
GetConfig() ([]byte, error)
WriteTmpConfig(tmpFileName string, configBytes []byte) error
CommitConfig(tmpFileName string) error
}
// Restart - Sends a message over channel to the go-routine
@@ -49,26 +71,26 @@ func (lc localAdminClient) Restart() error {
}
// ListLocks - Fetches lock information from local lock instrumentation.
func (lc localAdminClient) ListLocks(bucket, prefix string, relTime time.Duration) ([]VolumeLockInfo, error) {
return listLocksInfo(bucket, prefix, relTime), nil
func (lc localAdminClient) ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
return listLocksInfo(bucket, prefix, duration), nil
}
// Restart - Sends restart command to remote server via RPC.
func (rc remoteAdminClient) Restart() error {
args := AuthRPCArgs{}
reply := AuthRPCReply{}
return rc.Call("Admin.Restart", &args, &reply)
return rc.Call(serviceRestartRPC, &args, &reply)
}
// ListLocks - Sends list locks command to remote server via RPC.
func (rc remoteAdminClient) ListLocks(bucket, prefix string, relTime time.Duration) ([]VolumeLockInfo, error) {
func (rc remoteAdminClient) ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
listArgs := ListLocksQuery{
bucket: bucket,
prefix: prefix,
relTime: relTime,
bucket: bucket,
prefix: prefix,
duration: duration,
}
var reply ListLocksReply
if err := rc.Call("Admin.ListLocks", &listArgs, &reply); err != nil {
if err := rc.Call(listLocksRPC, &listArgs, &reply); err != nil {
return nil, err
}
return reply.volLocks, nil
@@ -85,7 +107,98 @@ func (lc localAdminClient) ReInitDisks() error {
func (rc remoteAdminClient) ReInitDisks() error {
args := AuthRPCArgs{}
reply := AuthRPCReply{}
return rc.Call("Admin.ReInitDisks", &args, &reply)
return rc.Call(reInitDisksRPC, &args, &reply)
}
// Uptime - Returns the uptime of this server. Timestamp is taken
// after object layer is initialized.
func (lc localAdminClient) Uptime() (time.Duration, error) {
if globalBootTime.IsZero() {
return time.Duration(0), errServerNotInitialized
}
return time.Now().UTC().Sub(globalBootTime), nil
}
// Uptime - returns the uptime of the server to which the RPC call is made.
func (rc remoteAdminClient) Uptime() (time.Duration, error) {
args := AuthRPCArgs{}
reply := UptimeReply{}
err := rc.Call(uptimeRPC, &args, &reply)
if err != nil {
return time.Duration(0), err
}
return reply.Uptime, nil
}
// GetConfig - returns config.json of the local server.
func (lc localAdminClient) GetConfig() ([]byte, error) {
if serverConfig == nil {
return nil, errors.New("config not present")
}
return json.Marshal(serverConfig)
}
// GetConfig - returns config.json of the remote server.
func (rc remoteAdminClient) GetConfig() ([]byte, error) {
args := AuthRPCArgs{}
reply := ConfigReply{}
if err := rc.Call(getConfigRPC, &args, &reply); err != nil {
return nil, err
}
return reply.Config, nil
}
// WriteTmpConfig - writes config file content to a temporary file on
// the local server.
func (lc localAdminClient) WriteTmpConfig(tmpFileName string, configBytes []byte) error {
return writeTmpConfigCommon(tmpFileName, configBytes)
}
// WriteTmpConfig - writes config file content to a temporary file on
// a remote node.
func (rc remoteAdminClient) WriteTmpConfig(tmpFileName string, configBytes []byte) error {
wArgs := WriteConfigArgs{
TmpFileName: tmpFileName,
Buf: configBytes,
}
err := rc.Call(writeTmpConfigRPC, &wArgs, &WriteConfigReply{})
if err != nil {
errorIf(err, "Failed to write temporary config file.")
return err
}
return nil
}
// CommitConfig - Move the new config in tmpFileName onto config.json
// on a local node.
func (lc localAdminClient) CommitConfig(tmpFileName string) error {
configFile := getConfigFile()
tmpConfigFile := filepath.Join(getConfigDir(), tmpFileName)
err := os.Rename(tmpConfigFile, configFile)
errorIf(err, fmt.Sprintf("Failed to rename %s to %s", tmpConfigFile, configFile))
return err
}
// CommitConfig - Move the new config in tmpFileName onto config.json
// on a remote node.
func (rc remoteAdminClient) CommitConfig(tmpFileName string) error {
cArgs := CommitConfigArgs{
FileName: tmpFileName,
}
cReply := CommitConfigReply{}
err := rc.Call(commitConfigRPC, &cArgs, &cReply)
if err != nil {
errorIf(err, "Failed to rename config file.")
return err
}
return nil
}
// adminPeer - represents an entity that implements Restart methods.
@@ -126,7 +239,7 @@ func makeAdminPeers(eps []*url.URL) adminPeers {
secretKey: serverCred.SecretKey,
serverAddr: ep.Host,
secureConn: globalIsSSL,
serviceEndpoint: path.Join(reservedBucket, adminPath),
serviceEndpoint: path.Join(minioReservedBucketPath, adminPath),
serviceName: "Admin",
}
@@ -175,8 +288,8 @@ func sendServiceCmd(cps adminPeers, cmd serviceSignal) {
}
// listPeerLocksInfo - fetch list of locks held on the given bucket,
// matching prefix older than relTime from all peer servers.
func listPeerLocksInfo(peers adminPeers, bucket, prefix string, relTime time.Duration) ([]VolumeLockInfo, error) {
// matching prefix held longer than duration from all peer servers.
func listPeerLocksInfo(peers adminPeers, bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
// Used to aggregate volume lock information from all nodes.
allLocks := make([][]VolumeLockInfo, len(peers))
errs := make([]error, len(peers))
@@ -188,11 +301,11 @@ func listPeerLocksInfo(peers adminPeers, bucket, prefix string, relTime time.Dur
go func(idx int, remotePeer adminPeer) {
defer wg.Done()
// `remotePeers` is right-shifted by one position relative to `peers`
allLocks[idx], errs[idx] = remotePeer.cmdRunner.ListLocks(bucket, prefix, relTime)
allLocks[idx], errs[idx] = remotePeer.cmdRunner.ListLocks(bucket, prefix, duration)
}(i+1, remotePeer)
}
wg.Wait()
allLocks[0], errs[0] = localPeer.cmdRunner.ListLocks(bucket, prefix, relTime)
allLocks[0], errs[0] = localPeer.cmdRunner.ListLocks(bucket, prefix, duration)
// Summarizing errors received for ListLocks RPC across all
// nodes. N B the possible unavailability of quorum in errors
@@ -241,3 +354,256 @@ func reInitPeerDisks(peers adminPeers) error {
wg.Wait()
return nil
}
// uptimeSlice - used to sort uptimes in chronological order.
type uptimeSlice []struct {
err error
uptime time.Duration
}
func (ts uptimeSlice) Len() int {
return len(ts)
}
func (ts uptimeSlice) Less(i, j int) bool {
return ts[i].uptime < ts[j].uptime
}
func (ts uptimeSlice) Swap(i, j int) {
ts[i], ts[j] = ts[j], ts[i]
}
// getPeerUptimes - returns the uptime since the last time read quorum
// was established on success. Otherwise returns errXLReadQuorum.
func getPeerUptimes(peers adminPeers) (time.Duration, error) {
// In a single node Erasure or FS backend setup the uptime of
// the setup is the uptime of the single minio server
// instance.
if !globalIsDistXL {
return time.Now().UTC().Sub(globalBootTime), nil
}
uptimes := make(uptimeSlice, len(peers))
// Get up time of all servers.
wg := sync.WaitGroup{}
for i, peer := range peers {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
uptimes[idx].uptime, uptimes[idx].err = peer.cmdRunner.Uptime()
}(i, peer)
}
wg.Wait()
// Sort uptimes in chronological order.
sort.Sort(uptimes)
// Pick the readQuorum'th uptime in chronological order. i.e,
// the time at which read quorum was (re-)established.
readQuorum := len(uptimes) / 2
validCount := 0
latestUptime := time.Duration(0)
for _, uptime := range uptimes {
if uptime.err != nil {
errorIf(uptime.err, "Unable to fetch uptime")
continue
}
validCount++
if validCount >= readQuorum {
latestUptime = uptime.uptime
break
}
}
// Less than readQuorum "Admin.Uptime" RPC call returned
// successfully, so read-quorum unavailable.
if validCount < readQuorum {
return time.Duration(0), InsufficientReadQuorum{}
}
return latestUptime, nil
}
// getPeerConfig - Fetches config.json from all nodes in the setup and
// returns the one that occurs in a majority of them.
func getPeerConfig(peers adminPeers) ([]byte, error) {
if !globalIsDistXL {
return peers[0].cmdRunner.GetConfig()
}
errs := make([]error, len(peers))
configs := make([][]byte, len(peers))
// Get config from all servers.
wg := sync.WaitGroup{}
for i, peer := range peers {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
configs[idx], errs[idx] = peer.cmdRunner.GetConfig()
}(i, peer)
}
wg.Wait()
// Find the maximally occurring config among peers in a
// distributed setup.
serverConfigs := make([]serverConfigV13, len(peers))
for i, configBytes := range configs {
if errs[i] != nil {
continue
}
// Unmarshal the received config files.
err := json.Unmarshal(configBytes, &serverConfigs[i])
if err != nil {
errorIf(err, "Failed to unmarshal serverConfig from ", peers[i].addr)
return nil, err
}
}
configJSON, err := getValidServerConfig(serverConfigs, errs)
if err != nil {
errorIf(err, "Unable to find a valid server config")
return nil, traceError(err)
}
// Return the config.json that was present quorum or more
// number of disks.
return json.Marshal(configJSON)
}
// getValidServerConfig - finds the server config that is present in
// quorum or more number of servers.
func getValidServerConfig(serverConfigs []serverConfigV13, errs []error) (serverConfigV13, error) {
// majority-based quorum
quorum := len(serverConfigs)/2 + 1
// Count the number of disks a config.json was found in.
configCounter := make([]int, len(serverConfigs))
// We group equal serverConfigs by the lowest index of the
// same value; e.g, let us take the following serverConfigs
// in a 4-node setup,
// serverConfigs == [c1, c2, c1, c1]
// configCounter == [3, 1, 0, 0]
// c1, c2 are the only distinct values that appear. c1 is
// identified by 0, the lowest index it appears in and c2 is
// identified by 1. So, we need to find the number of times
// each of these distinct values occur.
// Invariants:
// 1. At the beginning of the i-th iteration, the number of
// unique configurations seen so far is equal to the number of
// non-zero counter values in config[:i].
// 2. At the beginning of the i-th iteration, the sum of
// elements of configCounter[:i] is equal to the number of
// non-error configurations seen so far.
// For each of the serverConfig ...
for i := range serverConfigs {
// Skip nodes where getConfig failed.
if errs[i] != nil {
continue
}
// Check if it is equal to any of the configurations
// seen so far. If j == i is reached then we have an
// unseen configuration.
for j := 0; j <= i; j++ {
if j < i && configCounter[j] == 0 {
// serverConfigs[j] is known to be
// equal to a value that was already
// seen. See example above for
// clarity.
continue
} else if j < i && reflect.DeepEqual(serverConfigs[i], serverConfigs[j]) {
// serverConfigs[i] is equal to
// serverConfigs[j], update
// serverConfigs[j]'s counter since it
// is the lower index.
configCounter[j]++
break
} else if j == i {
// serverConfigs[i] is equal to no
// other value seen before. It is
// unique so far.
configCounter[i] = 1
break
} // else invariants specified above are violated.
}
}
// We find the maximally occurring server config and check if
// there is quorum.
var configJSON serverConfigV13
maxOccurrence := 0
for i, count := range configCounter {
if maxOccurrence < count {
maxOccurrence = count
configJSON = serverConfigs[i]
}
}
// If quorum nodes don't agree.
if maxOccurrence < quorum {
return serverConfigV13{}, errXLWriteQuorum
}
return configJSON, nil
}
// Write config contents into a temporary file on all nodes.
func writeTmpConfigPeers(peers adminPeers, tmpFileName string, configBytes []byte) []error {
// For a single-node minio server setup.
if !globalIsDistXL {
err := peers[0].cmdRunner.WriteTmpConfig(tmpFileName, configBytes)
return []error{err}
}
errs := make([]error, len(peers))
// Write config into temporary file on all nodes.
wg := sync.WaitGroup{}
for i, peer := range peers {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
errs[idx] = peer.cmdRunner.WriteTmpConfig(tmpFileName, configBytes)
}(i, peer)
}
wg.Wait()
// Return bytes written and errors (if any) during writing
// temporary config file.
return errs
}
// Move config contents from the given temporary file onto config.json
// on all nodes.
func commitConfigPeers(peers adminPeers, tmpFileName string) []error {
// For a single-node minio server setup.
if !globalIsDistXL {
return []error{peers[0].cmdRunner.CommitConfig(tmpFileName)}
}
errs := make([]error, len(peers))
// Rename temporary config file into configDir/config.json on
// all nodes.
wg := sync.WaitGroup{}
for i, peer := range peers {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
errs[idx] = peer.cmdRunner.CommitConfig(tmpFileName)
}(i, peer)
}
wg.Wait()
// Return errors (if any) received during rename.
return errs
}
+260
View File
@@ -0,0 +1,260 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"encoding/json"
"reflect"
"testing"
)
var (
config1 = []byte(`{
"version": "13",
"credential": {
"accessKey": "minio",
"secretKey": "minio123"
},
"region": "us-east-1",
"logger": {
"console": {
"enable": true,
"level": "debug"
},
"file": {
"enable": false,
"fileName": "",
"level": ""
}
},
"notify": {
"amqp": {
"1": {
"enable": false,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"streaming": {
"enable": false,
"clusterID": "",
"clientID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"elasticsearch": {
"1": {
"enable": false,
"url": "",
"index": ""
}
},
"redis": {
"1": {
"enable": false,
"address": "",
"password": "",
"key": ""
}
},
"postgresql": {
"1": {
"enable": false,
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": ""
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
}
}
}
}
`)
// diff from config1 - amqp.Enable is True
config2 = []byte(`{
"version": "13",
"credential": {
"accessKey": "minio",
"secretKey": "minio123"
},
"region": "us-east-1",
"logger": {
"console": {
"enable": true,
"level": "debug"
},
"file": {
"enable": false,
"fileName": "",
"level": ""
}
},
"notify": {
"amqp": {
"1": {
"enable": true,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"streaming": {
"enable": false,
"clusterID": "",
"clientID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"elasticsearch": {
"1": {
"enable": false,
"url": "",
"index": ""
}
},
"redis": {
"1": {
"enable": false,
"address": "",
"password": "",
"key": ""
}
},
"postgresql": {
"1": {
"enable": false,
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": ""
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
}
}
}
}
`)
)
// TestGetValidServerConfig - test for getValidServerConfig.
func TestGetValidServerConfig(t *testing.T) {
var c1, c2 serverConfigV13
err := json.Unmarshal(config1, &c1)
if err != nil {
t.Fatalf("json unmarshal of %s failed: %v", string(config1), err)
}
err = json.Unmarshal(config2, &c2)
if err != nil {
t.Fatalf("json unmarshal of %s failed: %v", string(config2), err)
}
// Valid config.
noErrs := []error{nil, nil, nil, nil}
serverConfigs := []serverConfigV13{c1, c2, c1, c1}
validConfig, err := getValidServerConfig(serverConfigs, noErrs)
if err != nil {
t.Errorf("Expected a valid config but received %v instead", err)
}
if !reflect.DeepEqual(validConfig, c1) {
t.Errorf("Expected valid config to be %v but received %v", config1, validConfig)
}
// Invalid config - no quorum.
serverConfigs = []serverConfigV13{c1, c2, c2, c1}
validConfig, err = getValidServerConfig(serverConfigs, noErrs)
if err != errXLWriteQuorum {
t.Errorf("Expected to fail due to lack of quorum but received %v", err)
}
// All errors
allErrs := []error{errDiskNotFound, errDiskNotFound, errDiskNotFound, errDiskNotFound}
serverConfigs = []serverConfigV13{{}, {}, {}, {}}
validConfig, err = getValidServerConfig(serverConfigs, allErrs)
if err != errXLWriteQuorum {
t.Errorf("Expected to fail due to lack of quorum but received %v", err)
}
}
+115 -5
View File
@@ -17,8 +17,13 @@
package cmd
import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net/rpc"
"os"
"path/filepath"
"time"
router "github.com/gorilla/mux"
@@ -37,9 +42,9 @@ type adminCmd struct {
// ListLocksQuery - wraps ListLocks API's query values to send over RPC.
type ListLocksQuery struct {
AuthRPCArgs
bucket string
prefix string
relTime time.Duration
bucket string
prefix string
duration time.Duration
}
// ListLocksReply - wraps ListLocks response over RPC.
@@ -48,6 +53,18 @@ type ListLocksReply struct {
volLocks []VolumeLockInfo
}
// UptimeReply - wraps the uptime response over RPC.
type UptimeReply struct {
AuthRPCReply
Uptime time.Duration
}
// ConfigReply - wraps the server config response over RPC.
type ConfigReply struct {
AuthRPCReply
Config []byte // json-marshalled bytes of serverConfigV13
}
// Restart - Restart this instance of minio server.
func (s *adminCmd) Restart(args *AuthRPCArgs, reply *AuthRPCReply) error {
if err := args.IsAuthenticated(); err != nil {
@@ -63,7 +80,7 @@ func (s *adminCmd) ListLocks(query *ListLocksQuery, reply *ListLocksReply) error
if err := query.IsAuthenticated(); err != nil {
return err
}
volLocks := listLocksInfo(query.bucket, query.prefix, query.relTime)
volLocks := listLocksInfo(query.bucket, query.prefix, query.duration)
*reply = ListLocksReply{volLocks: volLocks}
return nil
}
@@ -105,6 +122,99 @@ func (s *adminCmd) ReInitDisks(args *AuthRPCArgs, reply *AuthRPCReply) error {
return nil
}
// Uptime - returns the time when object layer was initialized on this server.
func (s *adminCmd) Uptime(args *AuthRPCArgs, reply *UptimeReply) error {
if err := args.IsAuthenticated(); err != nil {
return err
}
if globalBootTime.IsZero() {
return errServerNotInitialized
}
// N B The uptime is computed assuming that the system time is
// monotonic. This is not the case in time pkg in Go, see
// https://github.com/golang/go/issues/12914. This is expected
// to be fixed by go1.9.
*reply = UptimeReply{
Uptime: time.Now().UTC().Sub(globalBootTime),
}
return nil
}
// GetConfig - returns the config.json of this server.
func (s *adminCmd) GetConfig(args *AuthRPCArgs, reply *ConfigReply) error {
if err := args.IsAuthenticated(); err != nil {
return err
}
if serverConfig == nil {
return errors.New("config not present")
}
jsonBytes, err := json.Marshal(serverConfig)
if err != nil {
return err
}
reply.Config = jsonBytes
return nil
}
// WriteConfigArgs - wraps the bytes to be written and temporary file name.
type WriteConfigArgs struct {
AuthRPCArgs
TmpFileName string
Buf []byte
}
// WriteConfigReply - wraps the result of a writing config into a temporary file.
// the remote node.
type WriteConfigReply struct {
AuthRPCReply
}
func writeTmpConfigCommon(tmpFileName string, configBytes []byte) error {
tmpConfigFile := filepath.Join(getConfigDir(), tmpFileName)
err := ioutil.WriteFile(tmpConfigFile, configBytes, 0666)
errorIf(err, fmt.Sprintf("Failed to write to temporary config file %s", tmpConfigFile))
return err
}
// WriteTmpConfig - writes the supplied config contents onto the
// supplied temporary file.
func (s *adminCmd) WriteTmpConfig(wArgs *WriteConfigArgs, wReply *WriteConfigReply) error {
if err := wArgs.IsAuthenticated(); err != nil {
return err
}
return writeTmpConfigCommon(wArgs.TmpFileName, wArgs.Buf)
}
// CommitConfigArgs - wraps the config file name that needs to be
// committed into config.json on this node.
type CommitConfigArgs struct {
AuthRPCArgs
FileName string
}
// CommitConfigReply - represents response to commit of config file on
// this node.
type CommitConfigReply struct {
AuthRPCReply
}
// CommitConfig - Renames the temporary file into config.json on this node.
func (s *adminCmd) CommitConfig(cArgs *CommitConfigArgs, cReply *CommitConfigReply) error {
configFile := getConfigFile()
tmpConfigFile := filepath.Join(getConfigDir(), cArgs.FileName)
err := os.Rename(tmpConfigFile, configFile)
errorIf(err, fmt.Sprintf("Failed to rename %s to %s", tmpConfigFile, configFile))
return err
}
// registerAdminRPCRouter - registers RPC methods for service status,
// stop and restart commands.
func registerAdminRPCRouter(mux *router.Router) error {
@@ -114,7 +224,7 @@ func registerAdminRPCRouter(mux *router.Router) error {
if err != nil {
return traceError(err)
}
adminRouter := mux.NewRoute().PathPrefix(reservedBucket).Subrouter()
adminRouter := mux.NewRoute().PathPrefix(minioReservedBucketPath).Subrouter()
adminRouter.Path(adminPath).Handler(adminRPCServer)
return nil
}
+107 -5
View File
@@ -17,6 +17,7 @@
package cmd
import (
"encoding/json"
"net/url"
"testing"
"time"
@@ -52,7 +53,7 @@ func testAdminCmd(cmd cmdType, t *testing.T) {
<-globalServiceSignalCh
}()
ga := AuthRPCArgs{AuthToken: reply.AuthToken, RequestTime: time.Now().UTC()}
ga := AuthRPCArgs{AuthToken: reply.AuthToken}
genReply := AuthRPCReply{}
switch cmd {
case restartCmd:
@@ -107,8 +108,7 @@ func TestReInitDisks(t *testing.T) {
}
authArgs := AuthRPCArgs{
AuthToken: reply.AuthToken,
RequestTime: time.Now().UTC(),
AuthToken: reply.AuthToken,
}
authReply := AuthRPCReply{}
@@ -133,8 +133,7 @@ func TestReInitDisks(t *testing.T) {
}
authArgs = AuthRPCArgs{
AuthToken: fsReply.AuthToken,
RequestTime: time.Now().UTC(),
AuthToken: fsReply.AuthToken,
}
authReply = AuthRPCReply{}
// Attempt ReInitDisks service on a FS backend.
@@ -144,3 +143,106 @@ func TestReInitDisks(t *testing.T) {
errUnsupportedBackend, err)
}
}
// TestGetConfig - Test for GetConfig admin RPC.
func TestGetConfig(t *testing.T) {
// Reset global variables to start afresh.
resetTestGlobals()
rootPath, err := newTestConfig("us-east-1")
if err != nil {
t.Fatalf("Unable to initialize server config. %s", err)
}
defer removeAll(rootPath)
adminServer := adminCmd{}
creds := serverConfig.GetCredential()
args := LoginRPCArgs{
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: time.Now().UTC(),
}
reply := LoginRPCReply{}
err = adminServer.Login(&args, &reply)
if err != nil {
t.Fatalf("Failed to login to admin server - %v", err)
}
authArgs := AuthRPCArgs{
AuthToken: reply.AuthToken,
}
configReply := ConfigReply{}
err = adminServer.GetConfig(&authArgs, &configReply)
if err != nil {
t.Errorf("Expected GetConfig to pass but failed with %v", err)
}
var config serverConfigV13
err = json.Unmarshal(configReply.Config, &config)
if err != nil {
t.Errorf("Expected json unmarshal to pass but failed with %v", err)
}
}
// TestWriteAndCommitConfig - test for WriteTmpConfig and CommitConfig
// RPC handler.
func TestWriteAndCommitConfig(t *testing.T) {
// Reset global variables to start afresh.
resetTestGlobals()
rootPath, err := newTestConfig("us-east-1")
if err != nil {
t.Fatalf("Unable to initialize server config. %s", err)
}
defer removeAll(rootPath)
adminServer := adminCmd{}
creds := serverConfig.GetCredential()
args := LoginRPCArgs{
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: time.Now().UTC(),
}
reply := LoginRPCReply{}
err = adminServer.Login(&args, &reply)
if err != nil {
t.Fatalf("Failed to login to admin server - %v", err)
}
// Write temporary config.
buf := []byte("hello")
tmpFileName := mustGetUUID()
wArgs := WriteConfigArgs{
AuthRPCArgs: AuthRPCArgs{
AuthToken: reply.AuthToken,
},
TmpFileName: tmpFileName,
Buf: buf,
}
err = adminServer.WriteTmpConfig(&wArgs, &WriteConfigReply{})
if err != nil {
t.Fatalf("Failed to write temporary config %v", err)
}
if err != nil {
t.Errorf("Expected to succeed but failed %v", err)
}
cArgs := CommitConfigArgs{
AuthRPCArgs: AuthRPCArgs{
AuthToken: reply.AuthToken,
},
FileName: tmpFileName,
}
cReply := CommitConfigReply{}
err = adminServer.CommitConfig(&cArgs, &cReply)
if err != nil {
t.Fatalf("Failed to commit config file %v", err)
}
}
+42 -2
View File
@@ -56,6 +56,8 @@ const (
ErrInvalidBucketName
ErrInvalidDigest
ErrInvalidRange
ErrInvalidCopyPartRange
ErrInvalidCopyPartRangeSource
ErrInvalidMaxKeys
ErrInvalidMaxUploads
ErrInvalidMaxParts
@@ -83,6 +85,7 @@ const (
ErrInvalidPartOrder
ErrAuthorizationHeaderMalformed
ErrMalformedPOSTRequest
ErrPOSTFileRequired
ErrSignatureVersionNotSupported
ErrBucketNotEmpty
ErrAllAccessDisabled
@@ -136,6 +139,7 @@ const (
ErrObjectExistsAsDirectory
ErrPolicyNesting
ErrInvalidObjectName
ErrInvalidResourceName
ErrServerNotInitialized
// Add new extended error codes here.
// Please open a https://github.com/minio/minio/issues before adding
@@ -143,6 +147,7 @@ const (
ErrAdminInvalidAccessKey
ErrAdminInvalidSecretKey
ErrAdminConfigNoQuorum
)
// error code to APIError structure, these fields carry respective
@@ -333,6 +338,11 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "The body of your POST request is not well-formed multipart/form-data.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrPOSTFileRequired: {
Code: "InvalidArgument",
Description: "POST requires exactly one file upload per request.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrSignatureVersionNotSupported: {
Code: "InvalidRequest",
Description: "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.",
@@ -483,7 +493,7 @@ var errorCodeResponse = map[APIErrorCode]APIError{
},
ErrInvalidDuration: {
Code: "InvalidDuration",
Description: "Relative duration provided in the request is invalid.",
Description: "Duration provided in the request is invalid.",
HTTPStatusCode: http.StatusBadRequest,
},
@@ -533,6 +543,16 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "Configurations overlap. Configurations on the same bucket cannot share a common event type.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidCopyPartRange: {
Code: "InvalidArgument",
Description: "The x-amz-copy-source-range value must be of the form bytes=first-last where first and last are the zero-based offsets of the first and last bytes to copy",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidCopyPartRangeSource: {
Code: "InvalidArgument",
Description: "Range specified is not valid for source object",
HTTPStatusCode: http.StatusBadRequest,
},
/// S3 extensions.
ErrContentSHA256Mismatch: {
@@ -569,7 +589,12 @@ var errorCodeResponse = map[APIErrorCode]APIError{
},
ErrInvalidObjectName: {
Code: "XMinioInvalidObjectName",
Description: "Object name contains unsupported characters. Unsupported characters are `^*|\\\"",
Description: "Object name contains unsupported characters.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidResourceName: {
Code: "XMinioInvalidResourceName",
Description: "Resource name contains bad components such as \"..\" or \".\".",
HTTPStatusCode: http.StatusBadRequest,
},
ErrServerNotInitialized: {
@@ -587,6 +612,11 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "The secret key is invalid.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrAdminConfigNoQuorum: {
Code: "XMinioAdminConfigNoQuorum",
Description: "Configuration update failed because server quorum was not met",
HTTPStatusCode: http.StatusServiceUnavailable,
},
// Add your error structure here.
}
@@ -606,6 +636,14 @@ func toAPIErrorCode(err error) (apiErr APIErrorCode) {
apiErr = ErrSignatureDoesNotMatch
case errContentSHA256Mismatch:
apiErr = ErrContentSHA256Mismatch
case errDataTooLarge:
apiErr = ErrEntityTooLarge
case errDataTooSmall:
apiErr = ErrEntityTooSmall
case errInvalidAccessKeyLength:
apiErr = ErrAdminInvalidAccessKey
case errInvalidSecretKeyLength:
apiErr = ErrAdminInvalidSecretKey
}
if apiErr != ErrNone {
@@ -660,6 +698,8 @@ func toAPIErrorCode(err error) (apiErr APIErrorCode) {
apiErr = ErrEntityTooLarge
case ObjectTooSmall:
apiErr = ErrEntityTooSmall
case NotImplemented:
apiErr = ErrNotImplemented
default:
apiErr = ErrInternalError
}
+4
View File
@@ -103,6 +103,10 @@ func TestAPIErrCode(t *testing.T) {
StorageFull{},
ErrStorageFull,
},
{
NotImplemented{},
ErrNotImplemented,
},
{
errSignatureMismatch,
ErrSignatureDoesNotMatch,
+23 -6
View File
@@ -166,12 +166,13 @@ type ListBucketsResponse struct {
// Upload container for in progress multipart upload
type Upload struct {
Key string
UploadID string `xml:"UploadId"`
Initiator Initiator
Owner Owner
StorageClass string
Initiated string
Key string
UploadID string `xml:"UploadId"`
Initiator Initiator
Owner Owner
StorageClass string
Initiated string
HealUploadInfo *HealObjectInfo `xml:"HealObjectInfo,omitempty"`
}
// CommonPrefix container for prefix response in ListObjectsResponse
@@ -208,6 +209,13 @@ type CopyObjectResponse struct {
ETag string // md5sum of the copied object.
}
// CopyObjectPartResponse container returns ETag and LastModified of the successfully copied object
type CopyObjectPartResponse struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CopyPartResult" json:"-"`
LastModified string // time string of format "2006-01-02T15:04:05.000Z"
ETag string // md5sum of the copied object part.
}
// Initiator inherit from Owner struct, fields are same
type Initiator Owner
@@ -399,6 +407,14 @@ func generateCopyObjectResponse(etag string, lastModified time.Time) CopyObjectR
}
}
// generates CopyObjectPartResponse from etag and lastModified time.
func generateCopyObjectPartResponse(etag string, lastModified time.Time) CopyObjectPartResponse {
return CopyObjectPartResponse{
ETag: "\"" + etag + "\"",
LastModified: lastModified.UTC().Format(timeFormatAMZLong),
}
}
// generates InitiateMultipartUploadResponse for given bucket, key and uploadID.
func generateInitiateMultipartUploadResponse(bucket, key, uploadID string) InitiateMultipartUploadResponse {
return InitiateMultipartUploadResponse{
@@ -473,6 +489,7 @@ func generateListMultipartUploadsResponse(bucket string, multipartsInfo ListMult
newUpload.UploadID = upload.UploadID
newUpload.Key = upload.Object
newUpload.Initiated = upload.Initiated.UTC().Format(timeFormatAMZLong)
newUpload.HealUploadInfo = upload.HealUploadInfo
listMultipartUploadsResponse.Uploads[index] = newUpload
}
return listMultipartUploadsResponse
+2
View File
@@ -40,6 +40,8 @@ func registerAPIRouter(mux *router.Router) {
// HeadObject
bucket.Methods("HEAD").Path("/{object:.+}").HandlerFunc(api.HeadObjectHandler)
// CopyObjectPart
bucket.Methods("PUT").Path("/{object:.+}").HeadersRegexp("X-Amz-Copy-Source", ".*?(\\/|%2F).*?").HandlerFunc(api.CopyObjectPartHandler).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
// PutObjectPart
bucket.Methods("PUT").Path("/{object:.+}").HandlerFunc(api.PutObjectPartHandler).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}")
// ListObjectPxarts
+5 -2
View File
@@ -63,7 +63,9 @@ func isRequestPostPolicySignatureV4(r *http.Request) bool {
// Verify if the request has AWS Streaming Signature Version '4'. This is only valid for 'PUT' operation.
func isRequestSignStreamingV4(r *http.Request) bool {
return r.Header.Get("x-amz-content-sha256") == streamingContentSHA256 && r.Method == httpPUT
return r.Header.Get("x-amz-content-sha256") == streamingContentSHA256 &&
r.Header.Get("content-encoding") == streamingContentEncoding &&
r.Method == httpPUT
}
// Authorization type.
@@ -125,7 +127,8 @@ func checkRequestAuthType(r *http.Request, bucket, policyAction, region string)
if reqAuthType == authTypeAnonymous && policyAction != "" {
// http://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
return enforceBucketPolicy(bucket, policyAction, r.URL)
return enforceBucketPolicy(bucket, policyAction, r.URL.Path,
r.Referer(), r.URL.Query())
}
// By default return ErrAccessDenied
+15 -9
View File
@@ -36,13 +36,14 @@ func TestGetRequestAuthType(t *testing.T) {
{
req: &http.Request{
URL: &url.URL{
Host: "localhost:9000",
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
},
Header: http.Header{
"Authorization": []string{"AWS4-HMAC-SHA256 <cred_string>"},
"X-Amz-Content-Sha256": []string{streamingContentSHA256},
"Content-Encoding": []string{streamingContentEncoding},
},
Method: "PUT",
},
@@ -53,7 +54,7 @@ func TestGetRequestAuthType(t *testing.T) {
{
req: &http.Request{
URL: &url.URL{
Host: "localhost:9000",
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
},
@@ -68,7 +69,7 @@ func TestGetRequestAuthType(t *testing.T) {
{
req: &http.Request{
URL: &url.URL{
Host: "localhost:9000",
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
},
@@ -83,7 +84,7 @@ func TestGetRequestAuthType(t *testing.T) {
{
req: &http.Request{
URL: &url.URL{
Host: "localhost:9000",
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
RawQuery: "X-Amz-Credential=EXAMPLEINVALIDEXAMPL%2Fs3%2F20160314%2Fus-east-1",
@@ -96,7 +97,7 @@ func TestGetRequestAuthType(t *testing.T) {
{
req: &http.Request{
URL: &url.URL{
Host: "localhost:9000",
Host: "127.0.0.1:9000",
Scheme: httpScheme,
Path: "/",
},
@@ -315,7 +316,12 @@ func TestIsReqAuthenticated(t *testing.T) {
}
defer removeAll(path)
serverConfig.SetCredential(credential{"myuser", "mypassword"})
creds, err := createCredential("myuser", "mypassword")
if err != nil {
t.Fatalf("unable create credential, %s", err)
}
serverConfig.SetCredential(creds)
// List of test cases for validating http request authentication.
testCases := []struct {
@@ -325,11 +331,11 @@ func TestIsReqAuthenticated(t *testing.T) {
// When request is nil, internal error is returned.
{nil, ErrInternalError},
// When request is unsigned, access denied is returned.
{mustNewRequest("GET", "http://localhost:9000", 0, nil, t), ErrAccessDenied},
{mustNewRequest("GET", "http://127.0.0.1:9000", 0, nil, t), ErrAccessDenied},
// When request is properly signed, but has bad Content-MD5 header.
{mustNewSignedRequest("PUT", "http://localhost:9000", 5, bytes.NewReader([]byte("hello")), t), ErrBadDigest},
{mustNewSignedRequest("PUT", "http://127.0.0.1:9000", 5, bytes.NewReader([]byte("hello")), t), ErrBadDigest},
// When request is properly signed, error is none.
{mustNewSignedRequest("GET", "http://localhost:9000", 0, nil, t), ErrNone},
{mustNewSignedRequest("GET", "http://127.0.0.1:9000", 0, nil, t), ErrNone},
}
// Validates all testcases.
+33 -5
View File
@@ -35,6 +35,19 @@ type authConfig struct {
secureConn bool // Make TLS connection to RPC server or not.
serviceName string // Service name of auth server.
disableReconnect bool // Disable reconnect on failure or not.
/// Retry configurable values.
// Each retry unit multiplicative, measured in time.Duration.
// This is the basic unit used for calculating backoffs.
retryUnit time.Duration
// Maximum retry duration i.e A caller would wait no more than this
// duration to continue their loop.
retryCap time.Duration
// Maximum retries an call authRPC client would do for a failed
// RPC call.
retryAttemptThreshold int
}
// AuthRPCClient is a authenticated RPC client which does authentication before doing Call().
@@ -47,6 +60,18 @@ type AuthRPCClient struct {
// newAuthRPCClient - returns a JWT based authenticated (go) rpc client, which does automatic reconnect.
func newAuthRPCClient(config authConfig) *AuthRPCClient {
// Check if retry params are set properly if not default them.
emptyDuration := time.Duration(int64(0))
if config.retryUnit == emptyDuration {
config.retryUnit = defaultRetryUnit
}
if config.retryCap == emptyDuration {
config.retryCap = defaultRetryCap
}
if config.retryAttemptThreshold == 0 {
config.retryAttemptThreshold = globalAuthRPCRetryThreshold
}
return &AuthRPCClient{
rpcClient: newRPCClient(config.serverAddr, config.serviceEndpoint, config.secureConn),
config: config,
@@ -86,13 +111,13 @@ func (authClient *AuthRPCClient) Login() (err error) {
// call makes a RPC call after logs into the server.
func (authClient *AuthRPCClient) call(serviceMethod string, args interface {
SetAuthToken(authToken string)
SetRequestTime(requestTime time.Time)
}, reply interface{}) (err error) {
// On successful login, execute RPC call.
if err = authClient.Login(); err == nil {
authClient.Lock()
// Set token and timestamp before the rpc call.
args.SetAuthToken(authClient.authToken)
args.SetRequestTime(time.Now().UTC())
authClient.Unlock()
// Do RPC call.
err = authClient.rpcClient.Call(serviceMethod, args, reply)
@@ -103,11 +128,14 @@ func (authClient *AuthRPCClient) call(serviceMethod string, args interface {
// Call executes RPC call till success or globalAuthRPCRetryThreshold on ErrShutdown.
func (authClient *AuthRPCClient) Call(serviceMethod string, args interface {
SetAuthToken(authToken string)
SetRequestTime(requestTime time.Time)
}, reply interface{}) (err error) {
// Done channel is used to close any lingering retry routine, as soon
// as this function returns.
doneCh := make(chan struct{})
defer close(doneCh)
for i := range newRetryTimer(time.Second, 30*time.Second, MaxJitter, doneCh) {
for i := range newRetryTimer(authClient.config.retryUnit, authClient.config.retryCap, doneCh) {
if err = authClient.call(serviceMethod, args, reply); err == rpc.ErrShutdown {
// As connection at server side is closed, close the rpc client.
authClient.Close()
@@ -115,7 +143,7 @@ func (authClient *AuthRPCClient) Call(serviceMethod string, args interface {
// Retry if reconnect is not disabled.
if !authClient.config.disableReconnect {
// Retry until threshold reaches.
if i < globalAuthRPCRetryThreshold {
if i < authClient.config.retryAttemptThreshold {
continue
}
}
+185
View File
@@ -0,0 +1,185 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"encoding/xml"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"strconv"
"time"
"github.com/Azure/azure-sdk-for-go/storage"
)
// AnonGetBucketInfo - Get bucket metadata from azure anonymously.
func (a AzureObjects) AnonGetBucketInfo(bucket string) (bucketInfo BucketInfo, err error) {
url, err := url.Parse(a.client.GetBlobURL(bucket, ""))
if err != nil {
return bucketInfo, azureToObjectError(traceError(err))
}
url.RawQuery = "restype=container"
resp, err := http.Head(url.String())
if err != nil {
return bucketInfo, azureToObjectError(traceError(err), bucket)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return bucketInfo, azureToObjectError(traceError(anonErrToObjectErr(resp.StatusCode, bucket)), bucket)
}
t, err := time.Parse(time.RFC1123, resp.Header.Get("Last-Modified"))
if err != nil {
return bucketInfo, traceError(err)
}
bucketInfo = BucketInfo{
Name: bucket,
Created: t,
}
return bucketInfo, nil
}
// AnonGetObject - SendGET request without authentication.
// This is needed when clients send GET requests on objects that can be downloaded without auth.
func (a AzureObjects) AnonGetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) (err error) {
url := a.client.GetBlobURL(bucket, object)
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return azureToObjectError(traceError(err), bucket, object)
}
if length > 0 && startOffset > 0 {
req.Header.Add("Range", fmt.Sprintf("bytes=%d-%d", startOffset, startOffset+length-1))
} else if startOffset > 0 {
req.Header.Add("Range", fmt.Sprintf("bytes=%d-", startOffset))
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return azureToObjectError(traceError(err), bucket, object)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusPartialContent && resp.StatusCode != http.StatusOK {
return azureToObjectError(traceError(anonErrToObjectErr(resp.StatusCode, bucket, object)), bucket, object)
}
_, err = io.Copy(writer, resp.Body)
return traceError(err)
}
// AnonGetObjectInfo - Send HEAD request without authentication and convert the
// result to ObjectInfo.
func (a AzureObjects) AnonGetObjectInfo(bucket, object string) (objInfo ObjectInfo, err error) {
resp, err := http.Head(a.client.GetBlobURL(bucket, object))
if err != nil {
return objInfo, azureToObjectError(traceError(err), bucket, object)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return objInfo, azureToObjectError(traceError(anonErrToObjectErr(resp.StatusCode, bucket, object)), bucket, object)
}
var contentLength int64
contentLengthStr := resp.Header.Get("Content-Length")
if contentLengthStr != "" {
contentLength, err = strconv.ParseInt(contentLengthStr, 0, 64)
if err != nil {
return objInfo, azureToObjectError(traceError(errUnexpected), bucket, object)
}
}
t, err := time.Parse(time.RFC1123, resp.Header.Get("Last-Modified"))
if err != nil {
return objInfo, traceError(err)
}
objInfo.ModTime = t
objInfo.Bucket = bucket
objInfo.UserDefined = make(map[string]string)
if resp.Header.Get("Content-Encoding") != "" {
objInfo.UserDefined["Content-Encoding"] = resp.Header.Get("Content-Encoding")
}
objInfo.UserDefined["Content-Type"] = resp.Header.Get("Content-Type")
objInfo.MD5Sum = resp.Header.Get("Etag")
objInfo.ModTime = t
objInfo.Name = object
objInfo.Size = contentLength
return
}
// AnonListObjects - Use Azure equivalent ListBlobs.
func (a AzureObjects) AnonListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (result ListObjectsInfo, err error) {
params := storage.ListBlobsParameters{
Prefix: prefix,
Marker: marker,
Delimiter: delimiter,
MaxResults: uint(maxKeys),
}
q := azureListBlobsGetParameters(params)
q.Set("restype", "container")
q.Set("comp", "list")
url, err := url.Parse(a.client.GetBlobURL(bucket, ""))
if err != nil {
return result, azureToObjectError(traceError(err))
}
url.RawQuery = q.Encode()
resp, err := http.Get(url.String())
if err != nil {
return result, azureToObjectError(traceError(err))
}
defer resp.Body.Close()
var listResp storage.BlobListResponse
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
return result, azureToObjectError(traceError(err))
}
err = xml.Unmarshal(data, &listResp)
if err != nil {
return result, azureToObjectError(traceError(err))
}
result.IsTruncated = listResp.NextMarker != ""
result.NextMarker = listResp.NextMarker
for _, object := range listResp.Blobs {
t, e := time.Parse(time.RFC1123, object.Properties.LastModified)
if e != nil {
continue
}
result.Objects = append(result.Objects, ObjectInfo{
Bucket: bucket,
Name: object.Name,
ModTime: t,
Size: object.Properties.ContentLength,
MD5Sum: object.Properties.Etag,
ContentType: object.Properties.ContentType,
ContentEncoding: object.Properties.ContentEncoding,
})
}
result.Prefixes = listResp.BlobPrefixes
return result, nil
}
+43
View File
@@ -0,0 +1,43 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
// HealBucket - Not relevant.
func (a AzureObjects) HealBucket(bucket string) error {
return traceError(NotImplemented{})
}
// ListBucketsHeal - Not relevant.
func (a AzureObjects) ListBucketsHeal() (buckets []BucketInfo, err error) {
return nil, traceError(NotImplemented{})
}
// HealObject - Not relevant.
func (a AzureObjects) HealObject(bucket, object string) error {
return traceError(NotImplemented{})
}
// ListObjectsHeal - Not relevant.
func (a AzureObjects) ListObjectsHeal(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, error) {
return ListObjectsInfo{}, traceError(NotImplemented{})
}
// ListUploadsHeal - Not relevant.
func (a AzureObjects) ListUploadsHeal(bucket, prefix, marker, uploadIDMarker,
delimiter string, maxUploads int) (ListMultipartsInfo, error) {
return ListMultipartsInfo{}, traceError(NotImplemented{})
}
+637
View File
@@ -0,0 +1,637 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"encoding/base64"
"encoding/hex"
"fmt"
"hash"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"sync"
"time"
"github.com/Azure/azure-sdk-for-go/storage"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/sha256-simd"
)
const globalAzureAPIVersion = "2016-05-31"
// To store metadata during NewMultipartUpload which will be used after
// CompleteMultipartUpload to call SetBlobMetadata.
type azureMultipartMetaInfo struct {
meta map[string]map[string]string
*sync.Mutex
}
// Return metadata map of the multipart object.
func (a *azureMultipartMetaInfo) get(key string) map[string]string {
a.Lock()
defer a.Unlock()
return a.meta[key]
}
// Set metadata map for the multipart object.
func (a *azureMultipartMetaInfo) set(key string, value map[string]string) {
a.Lock()
defer a.Unlock()
a.meta[key] = value
}
// Delete metadata map for the multipart object.
func (a *azureMultipartMetaInfo) del(key string) {
a.Lock()
defer a.Unlock()
delete(a.meta, key)
}
// AzureObjects - Implements Object layer for Azure blob storage.
type AzureObjects struct {
client storage.BlobStorageClient // Azure sdk client
metaInfo azureMultipartMetaInfo
}
// Convert azure errors to minio object layer errors.
func azureToObjectError(err error, params ...string) error {
if err == nil {
return nil
}
e, ok := err.(*Error)
if !ok {
// Code should be fixed if this function is called without doing traceError()
// Else handling different situations in this function makes this function complicated.
errorIf(err, "Expected type *Error")
return err
}
err = e.e
bucket := ""
object := ""
if len(params) >= 1 {
bucket = params[0]
}
if len(params) == 2 {
object = params[1]
}
azureErr, ok := err.(storage.AzureStorageServiceError)
if !ok {
// We don't interpret non Azure errors. As azure errors will
// have StatusCode to help to convert to object errors.
return e
}
switch azureErr.Code {
case "ContainerAlreadyExists":
err = BucketExists{Bucket: bucket}
case "InvalidResourceName":
err = BucketNameInvalid{Bucket: bucket}
default:
switch azureErr.StatusCode {
case http.StatusNotFound:
if object != "" {
err = ObjectNotFound{bucket, object}
} else {
err = BucketNotFound{Bucket: bucket}
}
case http.StatusBadRequest:
err = BucketNameInvalid{Bucket: bucket}
}
}
e.e = err
return e
}
// Inits azure blob storage client and returns AzureObjects.
func newAzureLayer(endPoint string, account, key string, secure bool) (GatewayLayer, error) {
if endPoint == "" {
endPoint = storage.DefaultBaseURL
}
c, err := storage.NewClient(account, key, endPoint, globalAzureAPIVersion, secure)
if err != nil {
return AzureObjects{}, err
}
return &AzureObjects{
client: c.GetBlobService(),
metaInfo: azureMultipartMetaInfo{
meta: make(map[string]map[string]string),
Mutex: &sync.Mutex{},
},
}, nil
}
// Shutdown - save any gateway metadata to disk
// if necessary and reload upon next restart.
func (a AzureObjects) Shutdown() error {
// TODO
return nil
}
// StorageInfo - Not relevant to Azure backend.
func (a AzureObjects) StorageInfo() StorageInfo {
return StorageInfo{}
}
// MakeBucket - Create a new container on azure backend.
func (a AzureObjects) MakeBucket(bucket string) error {
err := a.client.CreateContainer(bucket, storage.ContainerAccessTypePrivate)
return azureToObjectError(traceError(err), bucket)
}
// GetBucketInfo - Get bucket metadata..
func (a AzureObjects) GetBucketInfo(bucket string) (BucketInfo, error) {
// Azure does not have an equivalent call, hence use ListContainers.
resp, err := a.client.ListContainers(storage.ListContainersParameters{
Prefix: bucket,
})
if err != nil {
return BucketInfo{}, azureToObjectError(traceError(err), bucket)
}
for _, container := range resp.Containers {
if container.Name == bucket {
t, e := time.Parse(time.RFC1123, container.Properties.LastModified)
if e == nil {
return BucketInfo{
Name: bucket,
Created: t,
}, nil
} // else continue
}
}
return BucketInfo{}, traceError(BucketNotFound{Bucket: bucket})
}
// ListBuckets - Lists all azure containers, uses Azure equivalent ListContainers.
func (a AzureObjects) ListBuckets() (buckets []BucketInfo, err error) {
resp, err := a.client.ListContainers(storage.ListContainersParameters{})
if err != nil {
return nil, azureToObjectError(traceError(err))
}
for _, container := range resp.Containers {
t, e := time.Parse(time.RFC1123, container.Properties.LastModified)
if e != nil {
return nil, traceError(e)
}
buckets = append(buckets, BucketInfo{
Name: container.Name,
Created: t,
})
}
return buckets, nil
}
// DeleteBucket - delete a container on azure, uses Azure equivalent DeleteContainer.
func (a AzureObjects) DeleteBucket(bucket string) error {
return azureToObjectError(traceError(a.client.DeleteContainer(bucket)), bucket)
}
// ListObjects - lists all blobs on azure with in a container filtered by prefix
// and marker, uses Azure equivalent ListBlobs.
func (a AzureObjects) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (result ListObjectsInfo, err error) {
resp, err := a.client.ListBlobs(bucket, storage.ListBlobsParameters{
Prefix: prefix,
Marker: marker,
Delimiter: delimiter,
MaxResults: uint(maxKeys),
})
if err != nil {
return result, azureToObjectError(traceError(err), bucket, prefix)
}
result.IsTruncated = resp.NextMarker != ""
result.NextMarker = resp.NextMarker
for _, object := range resp.Blobs {
t, e := time.Parse(time.RFC1123, object.Properties.LastModified)
if e != nil {
continue
}
result.Objects = append(result.Objects, ObjectInfo{
Bucket: bucket,
Name: object.Name,
ModTime: t,
Size: object.Properties.ContentLength,
MD5Sum: canonicalizeETag(object.Properties.Etag),
ContentType: object.Properties.ContentType,
ContentEncoding: object.Properties.ContentEncoding,
})
}
result.Prefixes = resp.BlobPrefixes
return result, nil
}
// GetObject - reads an object from azure. Supports additional
// parameters like offset and length which are synonymous with
// HTTP Range requests.
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (a AzureObjects) GetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) error {
byteRange := fmt.Sprintf("%d-", startOffset)
if length > 0 && startOffset > 0 {
byteRange = fmt.Sprintf("%d-%d", startOffset, startOffset+length-1)
}
var rc io.ReadCloser
var err error
if startOffset == 0 && length == 0 {
rc, err = a.client.GetBlob(bucket, object)
} else {
rc, err = a.client.GetBlobRange(bucket, object, byteRange, nil)
}
if err != nil {
return azureToObjectError(traceError(err), bucket, object)
}
_, err = io.Copy(writer, rc)
rc.Close()
return traceError(err)
}
// GetObjectInfo - reads blob metadata properties and replies back ObjectInfo,
// uses zure equivalent GetBlobProperties.
func (a AzureObjects) GetObjectInfo(bucket, object string) (objInfo ObjectInfo, err error) {
prop, err := a.client.GetBlobProperties(bucket, object)
if err != nil {
return objInfo, azureToObjectError(traceError(err), bucket, object)
}
t, err := time.Parse(time.RFC1123, prop.LastModified)
if err != nil {
return objInfo, traceError(err)
}
objInfo = ObjectInfo{
Bucket: bucket,
UserDefined: make(map[string]string),
MD5Sum: canonicalizeETag(prop.Etag),
ModTime: t,
Name: object,
Size: prop.ContentLength,
}
if prop.ContentEncoding != "" {
objInfo.UserDefined["Content-Encoding"] = prop.ContentEncoding
}
objInfo.UserDefined["Content-Type"] = prop.ContentType
return objInfo, nil
}
// Canonicalize the metadata headers, without this azure-sdk calculates
// incorrect signature. This attempt to canonicalize is to convert
// any HTTP header which is of form say `accept-encoding` should be
// converted to `Accept-Encoding` in its canonical form.
func canonicalMetadata(metadata map[string]string) (canonical map[string]string) {
canonical = make(map[string]string)
for k, v := range metadata {
canonical[http.CanonicalHeaderKey(k)] = v
}
return canonical
}
// PutObject - Create a new blob with the incoming data,
// uses Azure equivalent CreateBlockBlobFromReader.
func (a AzureObjects) PutObject(bucket, object string, size int64, data io.Reader, metadata map[string]string, sha256sum string) (objInfo ObjectInfo, err error) {
var sha256Writer hash.Hash
teeReader := data
if sha256sum != "" {
sha256Writer = sha256.New()
teeReader = io.TeeReader(data, sha256Writer)
}
delete(metadata, "md5Sum")
err = a.client.CreateBlockBlobFromReader(bucket, object, uint64(size), teeReader, canonicalMetadata(metadata))
if err != nil {
return objInfo, azureToObjectError(traceError(err), bucket, object)
}
if sha256sum != "" {
newSHA256sum := hex.EncodeToString(sha256Writer.Sum(nil))
if newSHA256sum != sha256sum {
a.client.DeleteBlob(bucket, object, nil)
return ObjectInfo{}, traceError(SHA256Mismatch{})
}
}
return a.GetObjectInfo(bucket, object)
}
// CopyObject - Copies a blob from source container to destination container.
// Uses Azure equivalent CopyBlob API.
func (a AzureObjects) CopyObject(srcBucket, srcObject, destBucket, destObject string, metadata map[string]string) (objInfo ObjectInfo, err error) {
err = a.client.CopyBlob(destBucket, destObject, a.client.GetBlobURL(srcBucket, srcObject))
if err != nil {
return objInfo, azureToObjectError(traceError(err), srcBucket, srcObject)
}
return a.GetObjectInfo(destBucket, destObject)
}
// DeleteObject - Deletes a blob on azure container, uses Azure
// equivalent DeleteBlob API.
func (a AzureObjects) DeleteObject(bucket, object string) error {
err := a.client.DeleteBlob(bucket, object, nil)
if err != nil {
return azureToObjectError(traceError(err), bucket, object)
}
return nil
}
// ListMultipartUploads - Incomplete implementation, for now just return the prefix if it is an incomplete upload.
// FIXME: Full ListMultipartUploads is not supported yet. It is supported just enough to help our client libs to
// support re-uploads. a.client.ListBlobs() can be made to return entries which include uncommitted blobs using
// which we need to filter out the committed blobs to get the list of uncommitted blobs.
func (a AzureObjects) ListMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error) {
result.MaxUploads = maxUploads
result.Prefix = prefix
result.Delimiter = delimiter
meta := a.metaInfo.get(prefix)
if meta == nil {
// In case minio was restarted after NewMultipartUpload and before CompleteMultipartUpload we expect
// the client to do a fresh upload so that any metadata like content-type are sent again in the
// NewMultipartUpload.
return result, nil
}
result.Uploads = []uploadMetadata{{prefix, prefix, time.Now().UTC(), "", nil}}
return result, nil
}
// NewMultipartUpload - Use Azure equivalent CreateBlockBlob.
func (a AzureObjects) NewMultipartUpload(bucket, object string, metadata map[string]string) (uploadID string, err error) {
// Azure doesn't return a unique upload ID and we use object name in place of it. Azure allows multiple uploads to
// co-exist as long as the user keeps the blocks uploaded (in block blobs) unique amongst concurrent upload attempts.
// Each concurrent client, keeps its own blockID list which it can commit.
uploadID = object
if metadata == nil {
// Store an empty map as a placeholder else ListObjectParts/PutObjectPart will not work properly.
metadata = make(map[string]string)
} else {
metadata = canonicalMetadata(metadata)
}
a.metaInfo.set(uploadID, metadata)
return uploadID, nil
}
// CopyObjectPart - Not implemented.
func (a AzureObjects) CopyObjectPart(srcBucket, srcObject, destBucket, destObject string, uploadID string, partID int, startOffset int64, length int64) (info PartInfo, err error) {
return info, traceError(NotImplemented{})
}
// Encode partID+md5Hex to a blockID.
func azureGetBlockID(partID int, md5Hex string) string {
return base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%.5d.%s", partID, md5Hex)))
}
// Decode blockID to partID+md5Hex.
func azureParseBlockID(blockID string) (int, string, error) {
idByte, err := base64.StdEncoding.DecodeString(blockID)
if err != nil {
return 0, "", traceError(err)
}
idStr := string(idByte)
splitRes := strings.Split(idStr, ".")
if len(splitRes) != 2 {
return 0, "", traceError(errUnexpected)
}
partID, err := strconv.Atoi(splitRes[0])
if err != nil {
return 0, "", traceError(err)
}
return partID, splitRes[1], nil
}
// PutObjectPart - Use Azure equivalent PutBlockWithLength.
func (a AzureObjects) PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string, sha256sum string) (info PartInfo, err error) {
if meta := a.metaInfo.get(uploadID); meta == nil {
return info, traceError(InvalidUploadID{})
}
var sha256Writer hash.Hash
if sha256sum != "" {
sha256Writer = sha256.New()
}
teeReader := io.TeeReader(data, sha256Writer)
id := azureGetBlockID(partID, md5Hex)
err = a.client.PutBlockWithLength(bucket, object, id, uint64(size), teeReader, nil)
if err != nil {
return info, azureToObjectError(traceError(err), bucket, object)
}
if sha256sum != "" {
newSHA256sum := hex.EncodeToString(sha256Writer.Sum(nil))
if newSHA256sum != sha256sum {
return PartInfo{}, traceError(SHA256Mismatch{})
}
}
info.PartNumber = partID
info.ETag = md5Hex
info.LastModified = time.Now().UTC()
info.Size = size
return info, nil
}
// ListObjectParts - Use Azure equivalent GetBlockList.
func (a AzureObjects) ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (result ListPartsInfo, err error) {
result.Bucket = bucket
result.Object = object
result.UploadID = uploadID
result.MaxParts = maxParts
if meta := a.metaInfo.get(uploadID); meta == nil {
return result, nil
}
resp, err := a.client.GetBlockList(bucket, object, storage.BlockListTypeUncommitted)
if err != nil {
return result, azureToObjectError(traceError(err), bucket, object)
}
tmpMaxParts := 0
partCount := 0 // Used for figuring out IsTruncated.
nextPartNumberMarker := 0
for _, part := range resp.UncommittedBlocks {
if tmpMaxParts == maxParts {
// Also takes care of the case if maxParts = 0
break
}
partCount++
partID, md5Hex, err := azureParseBlockID(part.Name)
if err != nil {
return result, err
}
if partID <= partNumberMarker {
continue
}
result.Parts = append(result.Parts, PartInfo{
partID,
time.Now().UTC(),
md5Hex,
part.Size,
})
tmpMaxParts++
nextPartNumberMarker = partID
}
if partCount < len(resp.UncommittedBlocks) {
result.IsTruncated = true
result.NextPartNumberMarker = nextPartNumberMarker
}
return result, nil
}
// AbortMultipartUpload - Not Implemented.
// There is no corresponding API in azure to abort an incomplete upload. The uncommmitted blocks
// gets deleted after one week.
func (a AzureObjects) AbortMultipartUpload(bucket, object, uploadID string) error {
a.metaInfo.del(uploadID)
return nil
}
// CompleteMultipartUpload - Use Azure equivalent PutBlockList.
func (a AzureObjects) CompleteMultipartUpload(bucket, object, uploadID string, uploadedParts []completePart) (objInfo ObjectInfo, err error) {
meta := a.metaInfo.get(uploadID)
if meta == nil {
return objInfo, traceError(InvalidUploadID{uploadID})
}
var blocks []storage.Block
for _, part := range uploadedParts {
blocks = append(blocks, storage.Block{
ID: azureGetBlockID(part.PartNumber, part.ETag),
Status: storage.BlockStatusUncommitted,
})
}
err = a.client.PutBlockList(bucket, object, blocks)
if err != nil {
return objInfo, azureToObjectError(traceError(err), bucket, object)
}
if len(meta) > 0 {
prop := storage.BlobHeaders{
ContentMD5: meta["Content-Md5"],
ContentLanguage: meta["Content-Language"],
ContentEncoding: meta["Content-Encoding"],
ContentType: meta["Content-Type"],
CacheControl: meta["Cache-Control"],
}
err = a.client.SetBlobProperties(bucket, object, prop)
if err != nil {
return objInfo, azureToObjectError(traceError(err), bucket, object)
}
}
a.metaInfo.del(uploadID)
return a.GetObjectInfo(bucket, object)
}
func anonErrToObjectErr(statusCode int, params ...string) error {
bucket := ""
object := ""
if len(params) >= 1 {
bucket = params[0]
}
if len(params) == 2 {
object = params[1]
}
switch statusCode {
case http.StatusNotFound:
if object != "" {
return ObjectNotFound{bucket, object}
}
return BucketNotFound{Bucket: bucket}
case http.StatusBadRequest:
if object != "" {
return ObjectNameInvalid{bucket, object}
}
return BucketNameInvalid{Bucket: bucket}
}
return errUnexpected
}
// Copied from github.com/Azure/azure-sdk-for-go/storage/blob.go
func azureListBlobsGetParameters(p storage.ListBlobsParameters) url.Values {
out := url.Values{}
if p.Prefix != "" {
out.Set("prefix", p.Prefix)
}
if p.Delimiter != "" {
out.Set("delimiter", p.Delimiter)
}
if p.Marker != "" {
out.Set("marker", p.Marker)
}
if p.Include != "" {
out.Set("include", p.Include)
}
if p.MaxResults != 0 {
out.Set("maxresults", fmt.Sprintf("%v", p.MaxResults))
}
if p.Timeout != 0 {
out.Set("timeout", fmt.Sprintf("%v", p.Timeout))
}
return out
}
// SetBucketPolicies - Azure supports three types of container policies:
// storage.ContainerAccessTypeContainer - readonly in minio terminology
// storage.ContainerAccessTypeBlob - readonly without listing in minio terminology
// storage.ContainerAccessTypePrivate - none in minio terminology
// As the common denominator for minio and azure is readonly and none, we support
// these two policies at the bucket level.
func (a AzureObjects) SetBucketPolicies(bucket string, policies []BucketAccessPolicy) error {
prefix := bucket + "/*" // For all objects inside the bucket.
if len(policies) != 1 {
return traceError(NotImplemented{})
}
if policies[0].Prefix != prefix {
return traceError(NotImplemented{})
}
if policies[0].Policy != policy.BucketPolicyReadOnly {
return traceError(NotImplemented{})
}
perm := storage.ContainerPermissions{
AccessType: storage.ContainerAccessTypeContainer,
AccessPolicies: nil,
}
err := a.client.SetContainerPermissions(bucket, perm, 0, "")
return azureToObjectError(traceError(err), bucket)
}
// GetBucketPolicies - Get the container ACL and convert it to canonical []bucketAccessPolicy
func (a AzureObjects) GetBucketPolicies(bucket string) ([]BucketAccessPolicy, error) {
perm, err := a.client.GetContainerPermissions(bucket, 0, "")
if err != nil {
return nil, azureToObjectError(traceError(err), bucket)
}
switch perm.AccessType {
case storage.ContainerAccessTypePrivate:
return nil, nil
case storage.ContainerAccessTypeContainer:
return []BucketAccessPolicy{{"", policy.BucketPolicyReadOnly}}, nil
}
return nil, azureToObjectError(traceError(NotImplemented{}))
}
// DeleteBucketPolicies - Set the container ACL to "private"
func (a AzureObjects) DeleteBucketPolicies(bucket string) error {
perm := storage.ContainerPermissions{
AccessType: storage.ContainerAccessTypePrivate,
AccessPolicies: nil,
}
err := a.client.SetContainerPermissions(bucket, perm, 0, "")
return azureToObjectError(traceError(err))
}
+142
View File
@@ -0,0 +1,142 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"net/http"
"reflect"
"testing"
"github.com/Azure/azure-sdk-for-go/storage"
)
// Test canonical metadata.
func TestCanonicalMetadata(t *testing.T) {
metadata := map[string]string{
"accept-encoding": "gzip",
"content-encoding": "gzip",
}
expectedCanonicalM := map[string]string{
"Accept-Encoding": "gzip",
"Content-Encoding": "gzip",
}
actualCanonicalM := canonicalMetadata(metadata)
if !reflect.DeepEqual(actualCanonicalM, expectedCanonicalM) {
t.Fatalf("Test failed, expected %#v, got %#v", expectedCanonicalM, actualCanonicalM)
}
}
// Add tests for azure to object error.
func TestAzureToObjectError(t *testing.T) {
testCases := []struct {
actualErr error
expectedErr error
bucket, object string
}{
{
nil, nil, "", "",
},
{
traceError(errUnexpected), errUnexpected, "", "",
},
{
traceError(errUnexpected), traceError(errUnexpected), "", "",
},
{
traceError(storage.AzureStorageServiceError{
Code: "ContainerAlreadyExists",
}), BucketExists{Bucket: "bucket"}, "bucket", "",
},
{
traceError(storage.AzureStorageServiceError{
Code: "InvalidResourceName",
}), BucketNameInvalid{Bucket: "bucket."}, "bucket.", "",
},
{
traceError(storage.AzureStorageServiceError{
StatusCode: http.StatusNotFound,
}), ObjectNotFound{
Bucket: "bucket",
Object: "object",
}, "bucket", "object",
},
{
traceError(storage.AzureStorageServiceError{
StatusCode: http.StatusNotFound,
}), BucketNotFound{Bucket: "bucket"}, "bucket", "",
},
{
traceError(storage.AzureStorageServiceError{
StatusCode: http.StatusBadRequest,
}), BucketNameInvalid{Bucket: "bucket."}, "bucket.", "",
},
}
for i, testCase := range testCases {
err := azureToObjectError(testCase.actualErr, testCase.bucket, testCase.object)
if err != nil {
if err.Error() != testCase.expectedErr.Error() {
t.Errorf("Test %d: Expected error %s, got %s", i+1, testCase.expectedErr, err)
}
}
}
}
// Test azureGetBlockID().
func TestAzureGetBlockID(t *testing.T) {
testCases := []struct {
partID int
md5 string
blockID string
}{
{1, "d41d8cd98f00b204e9800998ecf8427e", "MDAwMDEuZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="},
{2, "a7fb6b7b36ee4ed66b5546fac4690273", "MDAwMDIuYTdmYjZiN2IzNmVlNGVkNjZiNTU0NmZhYzQ2OTAyNzM="},
}
for _, test := range testCases {
blockID := azureGetBlockID(test.partID, test.md5)
if blockID != test.blockID {
t.Fatalf("%s is not equal to %s", blockID, test.blockID)
}
}
}
// Test azureParseBlockID().
func TestAzureParseBlockID(t *testing.T) {
testCases := []struct {
partID int
md5 string
blockID string
}{
{1, "d41d8cd98f00b204e9800998ecf8427e", "MDAwMDEuZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="},
{2, "a7fb6b7b36ee4ed66b5546fac4690273", "MDAwMDIuYTdmYjZiN2IzNmVlNGVkNjZiNTU0NmZhYzQ2OTAyNzM="},
}
for _, test := range testCases {
partID, md5, err := azureParseBlockID(test.blockID)
if err != nil {
t.Fatal(err)
}
if partID != test.partID {
t.Fatalf("%d not equal to %d", partID, test.partID)
}
if md5 != test.md5 {
t.Fatalf("%s not equal to %s", md5, test.md5)
}
}
_, _, err := azureParseBlockID("junk")
if err == nil {
t.Fatal("Expected azureParseBlockID() to return error")
}
}
+5 -20
View File
@@ -28,25 +28,9 @@ import (
humanize "github.com/dustin/go-humanize"
)
// Prepare benchmark backend
// Prepare XL/FS backend for benchmark.
func prepareBenchmarkBackend(instanceType string) (ObjectLayer, []string, error) {
switch instanceType {
// Total number of disks for FS backend is set to 1.
case FSTestStr:
obj, disk, err := prepareFS()
if err != nil {
return nil, nil, err
}
return obj, []string{disk}, nil
// Total number of disks for XL backend is set to 16.
case XLTestStr:
return prepareXL()
}
obj, disk, err := prepareFS()
if err != nil {
return nil, nil, err
}
return obj, []string{disk}, nil
return prepareTestBackend(instanceType)
}
// Benchmark utility functions for ObjectLayer.PutObject().
@@ -133,11 +117,12 @@ func runPutObjectPartBenchmark(b *testing.B, obj ObjectLayer, partSize int) {
}
metadata := make(map[string]string)
metadata["md5Sum"] = getMD5Hash([]byte(textPartData))
md5Sum, err = obj.PutObjectPart(bucket, object, uploadID, j, int64(len(textPartData)), bytes.NewBuffer(textPartData), metadata["md5Sum"], sha256sum)
var partInfo PartInfo
partInfo, err = obj.PutObjectPart(bucket, object, uploadID, j, int64(len(textPartData)), bytes.NewBuffer(textPartData), metadata["md5Sum"], sha256sum)
if err != nil {
b.Fatal(err)
}
if md5Sum != metadata["md5Sum"] {
if partInfo.ETag != metadata["md5Sum"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, md5Sum, metadata["md5Sum"])
}
}
+7 -2
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,6 +17,7 @@
package cmd
import (
"fmt"
"path"
"sync"
"time"
@@ -63,6 +64,10 @@ func (br *browserPeerAPIHandlers) SetAuthPeer(args SetAuthPeerArgs, reply *AuthR
return err
}
if !args.Creds.IsValid() {
return fmt.Errorf("Invalid credential passed")
}
// Update credentials in memory
serverConfig.SetCredential(args.Creds)
@@ -107,7 +112,7 @@ func updateCredsOnPeers(creds credential) map[string]error {
secretKey: serverCred.SecretKey,
serverAddr: peers[ix],
secureConn: globalIsSSL,
serviceEndpoint: path.Join(reservedBucket, browserPeerPath),
serviceEndpoint: path.Join(minioReservedBucketPath, browserPeerPath),
serviceName: "BrowserPeer",
})
+5 -5
View File
@@ -36,7 +36,7 @@ func (s *TestRPCBrowserPeerSuite) SetUpSuite(c *testing.T) {
serverAddr: s.testServer.Server.Listener.Addr().String(),
accessKey: s.testServer.AccessKey,
secretKey: s.testServer.SecretKey,
serviceEndpoint: path.Join(reservedBucket, browserPeerPath),
serviceEndpoint: path.Join(minioReservedBucketPath, browserPeerPath),
serviceName: "BrowserPeer",
}
}
@@ -63,9 +63,9 @@ func TestBrowserPeerRPC(t *testing.T) {
// Tests for browser peer rpc.
func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
// Construct RPC call arguments.
creds := credential{
AccessKey: "abcd1",
SecretKey: "abcd1234",
creds, err := createCredential("abcd1", "abcd1234")
if err != nil {
t.Fatalf("unable to create credential. %v", err)
}
// Validate for invalid token.
@@ -73,7 +73,7 @@ func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
args.AuthToken = "garbage"
rclient := newRPCClient(s.testAuthConf.serverAddr, s.testAuthConf.serviceEndpoint, false)
defer rclient.Close()
err := rclient.Call("BrowserPeer.SetAuthPeer", &args, &AuthRPCReply{})
err = rclient.Call("BrowserPeer.SetAuthPeer", &args, &AuthRPCReply{})
if err != nil {
if err.Error() != errInvalidToken.Error() {
t.Fatal(err)
+1 -1
View File
@@ -45,7 +45,7 @@ func registerBrowserPeerRPCRouter(mux *router.Router) error {
return traceError(err)
}
bpRouter := mux.NewRoute().PathPrefix(reservedBucket).Subrouter()
bpRouter := mux.NewRoute().PathPrefix(minioReservedBucketPath).Subrouter()
bpRouter.Path(browserPeerPath).Handler(bpRPCServer)
return nil
}
+1 -2
View File
@@ -18,7 +18,6 @@ package cmd
import (
"net/http"
"strings"
"github.com/gorilla/mux"
)
@@ -44,7 +43,7 @@ func validateListObjectsArgs(prefix, marker, delimiter string, maxKeys int) APIE
// Marker is set validate pre-condition.
if marker != "" {
// Marker not common with prefix is not implemented.
if !strings.HasPrefix(marker, prefix) {
if !hasPrefix(marker, prefix) {
return ErrNotImplemented
}
}
+109 -74
View File
@@ -19,7 +19,6 @@ package cmd
import (
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"net/http"
"net/url"
@@ -33,7 +32,7 @@ import (
// http://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
// Enforces bucket policies for a bucket for a given tatusaction.
func enforceBucketPolicy(bucket string, action string, reqURL *url.URL) (s3Error APIErrorCode) {
func enforceBucketPolicy(bucket, action, resource, referer string, queryParams url.Values) (s3Error APIErrorCode) {
// Verify if bucket actually exists
if err := checkBucketExist(bucket, newObjectLayerFn()); err != nil {
err = errorCause(err)
@@ -50,6 +49,10 @@ func enforceBucketPolicy(bucket string, action string, reqURL *url.URL) (s3Error
return ErrInternalError
}
if globalBucketPolicies == nil {
return ErrAccessDenied
}
// Fetch bucket policy, if policy is not set return access denied.
policy := globalBucketPolicies.GetBucketPolicy(bucket)
if policy == nil {
@@ -57,16 +60,21 @@ func enforceBucketPolicy(bucket string, action string, reqURL *url.URL) (s3Error
}
// Construct resource in 'arn:aws:s3:::examplebucket/object' format.
resource := bucketARNPrefix + strings.TrimSuffix(strings.TrimPrefix(reqURL.Path, "/"), "/")
arn := bucketARNPrefix + strings.TrimSuffix(strings.TrimPrefix(resource, "/"), "/")
// Get conditions for policy verification.
conditionKeyMap := make(map[string]set.StringSet)
for queryParam := range reqURL.Query() {
conditionKeyMap[queryParam] = set.CreateStringSet(reqURL.Query().Get(queryParam))
for queryParam := range queryParams {
conditionKeyMap[queryParam] = set.CreateStringSet(queryParams.Get(queryParam))
}
// Add request referer to conditionKeyMap if present.
if referer != "" {
conditionKeyMap["referer"] = set.CreateStringSet(referer)
}
// Validate action, resource and conditions with current policy statements.
if !bucketPolicyEvalStatements(action, resource, conditionKeyMap, policy.Statements) {
if !bucketPolicyEvalStatements(action, arn, conditionKeyMap, policy.Statements) {
return ErrAccessDenied
}
return ErrNone
@@ -81,10 +89,7 @@ func isBucketActionAllowed(action, bucket, prefix string) bool {
resource := bucketARNPrefix + path.Join(bucket, prefix)
var conditionKeyMap map[string]set.StringSet
// Validate action, resource and conditions with current policy statements.
if !bucketPolicyEvalStatements(action, resource, conditionKeyMap, policy.Statements) {
return false
}
return true
return bucketPolicyEvalStatements(action, resource, conditionKeyMap, policy.Statements)
}
// GetBucketLocationHandler - GET Bucket location.
@@ -160,7 +165,7 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
}
if keyMarker != "" {
// Marker not common with prefix is not implemented.
if !strings.HasPrefix(keyMarker, prefix) {
if !hasPrefix(keyMarker, prefix) {
writeErrorResponse(w, ErrNotImplemented, r.URL)
return
}
@@ -272,7 +277,11 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
for index, object := range deleteObjects.Objects {
wg.Add(1)
go func(i int, obj ObjectIdentifier) {
objectLock := globalNSMutex.NewNSLock(bucket, obj.ObjectName)
objectLock.Lock()
defer objectLock.Unlock()
defer wg.Done()
dErr := objectAPI.DeleteObject(bucket, obj.ObjectName)
if dErr != nil {
dErrs[i] = dErr
@@ -321,9 +330,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
ObjInfo: ObjectInfo{
Name: dobj.ObjectName,
},
ReqParams: map[string]string{
"sourceIPAddress": r.RemoteAddr,
},
ReqParams: extractReqParams(r),
})
}
}
@@ -388,6 +395,13 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
// Require Content-Length to be set in the request
size := r.ContentLength
if size < 0 {
writeErrorResponse(w, ErrMissingContentLength, r.URL)
return
}
// Here the parameter is the size of the form data that should
// be loaded in memory, the remaining being put in temporary files.
reader, err := r.MultipartReader()
@@ -397,21 +411,54 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
fileBody, fileName, formValues, err := extractPostPolicyFormValues(reader)
// Read multipart data and save in memory and in the disk if needed
form, err := reader.ReadForm(maxFormMemory)
if err != nil {
errorIf(err, "Unable to initialize multipart reader.")
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
}
// Remove all tmp files creating during multipart upload
defer form.RemoveAll()
// Extract all form fields
fileBody, fileName, fileSize, formValues, err := extractPostPolicyFormValues(form)
if err != nil {
errorIf(err, "Unable to parse form values.")
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
}
bucket := mux.Vars(r)["bucket"]
formValues["Bucket"] = bucket
if fileName != "" && strings.Contains(formValues["Key"], "${filename}") {
// Check if file is provided, error out otherwise.
if fileBody == nil {
writeErrorResponse(w, ErrPOSTFileRequired, r.URL)
return
}
// Close multipart file
defer fileBody.Close()
bucket := mux.Vars(r)["bucket"]
formValues.Set("Bucket", bucket)
if fileName != "" && strings.Contains(formValues.Get("Key"), "${filename}") {
// S3 feature to replace ${filename} found in Key form field
// by the filename attribute passed in multipart
formValues["Key"] = strings.Replace(formValues["Key"], "${filename}", fileName, -1)
formValues.Set("Key", strings.Replace(formValues.Get("Key"), "${filename}", fileName, -1))
}
object := formValues.Get("Key")
successRedirect := formValues.Get("success_action_redirect")
successStatus := formValues.Get("success_action_status")
var redirectURL *url.URL
if successRedirect != "" {
redirectURL, err = url.Parse(successRedirect)
if err != nil {
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
}
}
object := formValues["Key"]
// Verify policy signature.
apiErr := doesPolicySignatureMatch(formValues)
@@ -420,7 +467,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
policyBytes, err := base64.StdEncoding.DecodeString(formValues["Policy"])
policyBytes, err := base64.StdEncoding.DecodeString(formValues.Get("Policy"))
if err != nil {
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
@@ -438,83 +485,71 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
// Use rangeReader to ensure that object size is within expected range.
// Ensure that the object size is within expected range, also the file size
// should not exceed the maximum single Put size (5 GiB)
lengthRange := postPolicyForm.Conditions.ContentLengthRange
if lengthRange.Valid {
// If policy restricted the size of the object.
fileBody = &rangeReader{
Reader: fileBody,
Min: lengthRange.Min,
Max: lengthRange.Max,
if fileSize < lengthRange.Min {
errorIf(err, "Unable to create object.")
writeErrorResponse(w, toAPIErrorCode(errDataTooSmall), r.URL)
return
}
} else {
// Default values of min/max size of the object.
fileBody = &rangeReader{
Reader: fileBody,
Min: 0,
Max: maxObjectSize,
if fileSize > lengthRange.Max || isMaxObjectSize(fileSize) {
errorIf(err, "Unable to create object.")
writeErrorResponse(w, toAPIErrorCode(errDataTooLarge), r.URL)
return
}
}
// Extract metadata to be saved from received Form.
metadata := extractMetadataFromForm(formValues)
sha256sum := ""
objectLock := globalNSMutex.NewNSLock(bucket, object)
objectLock.Lock()
defer objectLock.Unlock()
objInfo, err := objectAPI.PutObject(bucket, object, -1, fileBody, metadata, sha256sum)
objInfo, err := objectAPI.PutObject(bucket, object, fileSize, fileBody, metadata, sha256sum)
if err != nil {
errorIf(err, "Unable to create object.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
w.Header().Set("ETag", "\""+objInfo.MD5Sum+"\"")
w.Header().Set("ETag", `"`+objInfo.MD5Sum+`"`)
w.Header().Set("Location", getObjectLocation(bucket, object))
successRedirect := formValues[http.CanonicalHeaderKey("success_action_redirect")]
successStatus := formValues[http.CanonicalHeaderKey("success_action_status")]
// Notify object created event.
defer eventNotify(eventData{
Type: ObjectCreatedPost,
Bucket: objInfo.Bucket,
ObjInfo: objInfo,
ReqParams: extractReqParams(r),
})
if successStatus == "" && successRedirect == "" {
writeSuccessNoContent(w)
} else {
if successRedirect != "" {
redirectURL := successRedirect + "?" + fmt.Sprintf("bucket=%s&key=%s&etag=%s",
bucket,
getURLEncodedName(object),
getURLEncodedName("\""+objInfo.MD5Sum+"\""))
writeRedirectSeeOther(w, redirectURL)
} else {
// Decide what http response to send depending on success_action_status parameter
switch successStatus {
case "201":
resp := encodeResponse(PostResponse{
Bucket: bucket,
Key: object,
ETag: "\"" + objInfo.MD5Sum + "\"",
Location: getObjectLocation(bucket, object),
})
writeResponse(w, http.StatusCreated, resp, "application/xml")
case "200":
writeSuccessResponseHeadersOnly(w)
default:
writeSuccessNoContent(w)
}
}
if successRedirect != "" {
// Replace raw query params..
redirectURL.RawQuery = getRedirectPostRawQuery(objInfo)
writeRedirectSeeOther(w, redirectURL.String())
return
}
// Notify object created event.
eventNotify(eventData{
Type: ObjectCreatedPost,
Bucket: bucket,
ObjInfo: objInfo,
ReqParams: map[string]string{
"sourceIPAddress": r.RemoteAddr,
},
})
// Decide what http response to send depending on success_action_status parameter
switch successStatus {
case "201":
resp := encodeResponse(PostResponse{
Bucket: objInfo.Bucket,
Key: objInfo.Name,
ETag: `"` + objInfo.MD5Sum + `"`,
Location: getObjectLocation(objInfo.Bucket, objInfo.Name),
})
writeResponse(w, http.StatusCreated, resp, "application/xml")
case "200":
writeSuccessResponseHeadersOnly(w)
default:
writeSuccessNoContent(w)
}
}
// HeadBucketHandler - HEAD Bucket
+1 -4
View File
@@ -209,10 +209,7 @@ func writeNotification(w http.ResponseWriter, notification map[string][]Notifica
_, err = w.Write(append(notificationBytes, crlf...))
// Make sure we have flushed, this would set Transfer-Encoding: chunked.
w.(http.Flusher).Flush()
if err != nil {
return err
}
return nil
return err
}
// CRLF character used for chunked transfer in accordance with HTTP standards.
+14 -14
View File
@@ -133,27 +133,27 @@ func isValidQueueID(queueARN string) bool {
// Is Queue identifier valid?.
if isAMQPQueue(sqsARN) { // AMQP eueue.
amqpN := serverConfig.GetAMQPNotifyByID(sqsARN.AccountID)
amqpN := serverConfig.Notify.GetAMQPByID(sqsARN.AccountID)
return amqpN.Enable && amqpN.URL != ""
} else if isNATSQueue(sqsARN) {
natsN := serverConfig.GetNATSNotifyByID(sqsARN.AccountID)
natsN := serverConfig.Notify.GetNATSByID(sqsARN.AccountID)
return natsN.Enable && natsN.Address != ""
} else if isElasticQueue(sqsARN) { // Elastic queue.
elasticN := serverConfig.GetElasticSearchNotifyByID(sqsARN.AccountID)
elasticN := serverConfig.Notify.GetElasticSearchByID(sqsARN.AccountID)
return elasticN.Enable && elasticN.URL != ""
} else if isRedisQueue(sqsARN) { // Redis queue.
redisN := serverConfig.GetRedisNotifyByID(sqsARN.AccountID)
redisN := serverConfig.Notify.GetRedisByID(sqsARN.AccountID)
return redisN.Enable && redisN.Addr != ""
} else if isPostgreSQLQueue(sqsARN) {
pgN := serverConfig.GetPostgreSQLNotifyByID(sqsARN.AccountID)
pgN := serverConfig.Notify.GetPostgreSQLByID(sqsARN.AccountID)
// Postgres can work with only default conn. info.
return pgN.Enable
} else if isKafkaQueue(sqsARN) {
kafkaN := serverConfig.GetKafkaNotifyByID(sqsARN.AccountID)
kafkaN := serverConfig.Notify.GetKafkaByID(sqsARN.AccountID)
return (kafkaN.Enable && len(kafkaN.Brokers) > 0 &&
kafkaN.Topic != "")
} else if isWebhookQueue(sqsARN) {
webhookN := serverConfig.GetWebhookNotifyByID(sqsARN.AccountID)
webhookN := serverConfig.Notify.GetWebhookByID(sqsARN.AccountID)
return webhookN.Enable && webhookN.Endpoint != ""
}
return false
@@ -253,19 +253,19 @@ func unmarshalSqsARN(queueARN string) (mSqs arnSQS) {
}
sqsType := strings.TrimPrefix(queueARN, minioSqs+serverConfig.GetRegion()+":")
switch {
case strings.HasSuffix(sqsType, queueTypeAMQP):
case hasSuffix(sqsType, queueTypeAMQP):
mSqs.Type = queueTypeAMQP
case strings.HasSuffix(sqsType, queueTypeNATS):
case hasSuffix(sqsType, queueTypeNATS):
mSqs.Type = queueTypeNATS
case strings.HasSuffix(sqsType, queueTypeElastic):
case hasSuffix(sqsType, queueTypeElastic):
mSqs.Type = queueTypeElastic
case strings.HasSuffix(sqsType, queueTypeRedis):
case hasSuffix(sqsType, queueTypeRedis):
mSqs.Type = queueTypeRedis
case strings.HasSuffix(sqsType, queueTypePostgreSQL):
case hasSuffix(sqsType, queueTypePostgreSQL):
mSqs.Type = queueTypePostgreSQL
case strings.HasSuffix(sqsType, queueTypeKafka):
case hasSuffix(sqsType, queueTypeKafka):
mSqs.Type = queueTypeKafka
case strings.HasSuffix(sqsType, queueTypeWebhook):
case hasSuffix(sqsType, queueTypeWebhook):
mSqs.Type = queueTypeWebhook
} // Add more queues here.
mSqs.AccountID = strings.TrimSuffix(sqsType, ":"+mSqs.Type)
+65 -14
View File
@@ -21,6 +21,8 @@ import (
"io"
"io/ioutil"
"net/http"
"runtime"
"strings"
humanize "github.com/dustin/go-humanize"
mux "github.com/gorilla/mux"
@@ -63,6 +65,10 @@ func bucketPolicyActionMatch(action string, statement policyStatement) bool {
// Match function matches wild cards in 'pattern' for resource.
func resourceMatch(pattern, resource string) bool {
if runtime.GOOS == "windows" {
// For windows specifically make sure we are case insensitive.
return wildcard.Match(strings.ToLower(pattern), strings.ToLower(resource))
}
return wildcard.Match(pattern, resource)
}
@@ -71,6 +77,10 @@ func actionMatch(pattern, action string) bool {
return wildcard.MatchSimple(pattern, action)
}
func refererMatch(pattern, referer string) bool {
return wildcard.MatchSimple(pattern, referer)
}
// Verify if given resource matches with policy statement.
func bucketPolicyResourceMatch(resource string, statement policyStatement) bool {
// the resource rule for object could contain "*" wild card.
@@ -85,33 +95,74 @@ func bucketPolicyConditionMatch(conditions map[string]set.StringSet, statement p
// Supports following conditions.
// - StringEquals
// - StringNotEquals
// - StringLike
// - StringNotLike
//
// Supported applicable condition keys for each conditions.
// - s3:prefix
// - s3:max-keys
var conditionMatches = true
// - s3:aws-Referer
// The following loop evaluates the logical AND of all the
// conditions in the statement. Note: we can break out of the
// loop if and only if a condition evaluates to false.
for condition, conditionKeyVal := range statement.Conditions {
prefixConditon := conditionKeyVal["s3:prefix"]
maxKeyCondition := conditionKeyVal["s3:max-keys"]
if condition == "StringEquals" {
if !conditionKeyVal["s3:prefix"].Equals(conditions["prefix"]) {
conditionMatches = false
break
// If there is no condition with "s3:prefix" or "s3:max-keys" condition key
// then there is nothing to check condition against.
if !prefixConditon.IsEmpty() && !prefixConditon.Equals(conditions["prefix"]) {
return false
}
if !conditionKeyVal["s3:max-keys"].Equals(conditions["max-keys"]) {
conditionMatches = false
break
if !maxKeyCondition.IsEmpty() && !maxKeyCondition.Equals(conditions["max-keys"]) {
return false
}
} else if condition == "StringNotEquals" {
if !conditionKeyVal["s3:prefix"].Equals(conditions["prefix"]) {
conditionMatches = false
break
// If there is no condition with "s3:prefix" or "s3:max-keys" condition key
// then there is nothing to check condition against.
if !prefixConditon.IsEmpty() && prefixConditon.Equals(conditions["prefix"]) {
return false
}
if !conditionKeyVal["s3:max-keys"].Equals(conditions["max-keys"]) {
conditionMatches = false
break
if !maxKeyCondition.IsEmpty() && maxKeyCondition.Equals(conditions["max-keys"]) {
return false
}
} else if condition == "StringLike" {
awsReferers := conditionKeyVal["aws:Referer"]
// Skip empty condition, it is trivially satisfied.
if awsReferers.IsEmpty() {
continue
}
// wildcard match of referer in statement was not empty.
// StringLike has a match, i.e, condition evaluates to true.
refererFound := false
for referer := range conditions["referer"] {
if !awsReferers.FuncMatch(refererMatch, referer).IsEmpty() {
refererFound = true
break
}
}
// No matching referer found, so the condition is false.
if !refererFound {
return false
}
} else if condition == "StringNotLike" {
awsReferers := conditionKeyVal["aws:Referer"]
// Skip empty condition, it is trivially satisfied.
if awsReferers.IsEmpty() {
continue
}
// wildcard match of referer in statement was not empty.
// StringNotLike has a match, i.e, condition evaluates to false.
for referer := range conditions["referer"] {
if !awsReferers.FuncMatch(refererMatch, referer).IsEmpty() {
return false
}
}
}
}
return conditionMatches
return true
}
// PutBucketPolicyHandler - PUT Bucket policy
+34 -5
View File
@@ -913,7 +913,7 @@ func TestBucketPolicyConditionMatch(t *testing.T) {
statementCondition: getStatementWithCondition("StringNotEquals", "s3:prefix", "Asia/"),
condition: getInnerMap("prefix", "Asia/"),
expectedMatch: true,
expectedMatch: false,
},
// Test case - 6.
// StringNotEquals condition doesn't match.
@@ -922,7 +922,7 @@ func TestBucketPolicyConditionMatch(t *testing.T) {
statementCondition: getStatementWithCondition("StringNotEquals", "s3:prefix", "Asia/"),
condition: getInnerMap("prefix", "Africa/"),
expectedMatch: false,
expectedMatch: true,
},
// Test case - 7.
// StringNotEquals condition matches.
@@ -931,7 +931,7 @@ func TestBucketPolicyConditionMatch(t *testing.T) {
statementCondition: getStatementWithCondition("StringNotEquals", "s3:max-keys", "Asia/"),
condition: getInnerMap("max-keys", "Asia/"),
expectedMatch: true,
expectedMatch: false,
},
// Test case - 8.
// StringNotEquals condition doesn't match.
@@ -940,7 +940,35 @@ func TestBucketPolicyConditionMatch(t *testing.T) {
statementCondition: getStatementWithCondition("StringNotEquals", "s3:max-keys", "Asia/"),
condition: getInnerMap("max-keys", "Africa/"),
expectedMatch: false,
expectedMatch: true,
},
// Test case - 9.
// StringLike condition matches.
{
statementCondition: getStatementWithCondition("StringLike", "aws:Referer", "http://www.example.com/"),
condition: getInnerMap("referer", "http://www.example.com/"),
expectedMatch: true,
},
// Test case - 10.
// StringLike condition doesn't match.
{
statementCondition: getStatementWithCondition("StringLike", "aws:Referer", "http://www.example.com/"),
condition: getInnerMap("referer", "www.somethingelse.com"),
expectedMatch: false,
},
// Test case - 11.
// StringNotLike condition evaluates to false.
{
statementCondition: getStatementWithCondition("StringNotLike", "aws:Referer", "http://www.example.com/"),
condition: getInnerMap("referer", "http://www.example.com/"),
expectedMatch: false,
},
// Test case - 12.
// StringNotLike condition evaluates to true.
{
statementCondition: getStatementWithCondition("StringNotLike", "aws:Referer", "http://www.example.com/"),
condition: getInnerMap("referer", "http://somethingelse.com/"),
expectedMatch: true,
},
}
@@ -949,7 +977,8 @@ func TestBucketPolicyConditionMatch(t *testing.T) {
// call the function under test and assert the result with the expected result.
doesMatch := bucketPolicyConditionMatch(tc.condition, tc.statementCondition)
if tc.expectedMatch != doesMatch {
t.Errorf("Expected the match to be `%v`; got `%v`.", tc.expectedMatch, doesMatch)
t.Errorf("Expected the match to be `%v`; got `%v` - %v %v.",
tc.expectedMatch, doesMatch, tc.condition, tc.statementCondition)
}
})
}
+28 -13
View File
@@ -29,17 +29,22 @@ import (
"github.com/minio/minio-go/pkg/set"
)
var conditionKeyActionMap = map[string]set.StringSet{
"s3:prefix": set.CreateStringSet("s3:ListBucket"),
"s3:max-keys": set.CreateStringSet("s3:ListBucket"),
}
// supportedActionMap - lists all the actions supported by minio.
var supportedActionMap = set.CreateStringSet("*", "s3:*", "s3:GetObject",
"s3:ListBucket", "s3:PutObject", "s3:GetBucketLocation", "s3:DeleteObject",
"s3:AbortMultipartUpload", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts")
// supported Conditions type.
var supportedConditionsType = set.CreateStringSet("StringEquals", "StringNotEquals")
var supportedConditionsType = set.CreateStringSet("StringEquals", "StringNotEquals", "StringLike", "StringNotLike")
// Validate s3:prefix, s3:max-keys are present if not
// supported keys for the conditions.
var supportedConditionsKey = set.CreateStringSet("s3:prefix", "s3:max-keys")
var supportedConditionsKey = set.CreateStringSet("s3:prefix", "s3:max-keys", "aws:Referer")
// supportedEffectMap - supported effects.
var supportedEffectMap = set.CreateStringSet("Allow", "Deny")
@@ -106,12 +111,12 @@ func isValidResources(resources set.StringSet) (err error) {
return err
}
for resource := range resources {
if !strings.HasPrefix(resource, bucketARNPrefix) {
if !hasPrefix(resource, bucketARNPrefix) {
err = errors.New("Unsupported resource style found: " + resource + ", please validate your policy document")
return err
}
resourceSuffix := strings.SplitAfter(resource, bucketARNPrefix)[1]
if len(resourceSuffix) == 0 || strings.HasPrefix(resourceSuffix, "/") {
if len(resourceSuffix) == 0 || hasPrefix(resourceSuffix, "/") {
err = errors.New("Invalid resource style found: " + resource + ", please validate your policy document")
return err
}
@@ -178,11 +183,12 @@ func isValidPrincipals(principal interface{}) (err error) {
return nil
}
// isValidConditions - are valid conditions.
func isValidConditions(conditions map[string]map[string]set.StringSet) (err error) {
// Verify conditions should be valid.
// Validate if stringEquals, stringNotEquals are present
// if not throw an error.
// isValidConditions - returns nil if the given conditions valid and
// corresponding error otherwise.
func isValidConditions(actions set.StringSet, conditions map[string]map[string]set.StringSet) (err error) {
// Verify conditions should be valid. Validate if only
// supported condition keys are present and return error
// otherwise.
conditionKeyVal := make(map[string]set.StringSet)
for conditionType := range conditions {
if !supportedConditionsType.Contains(conditionType) {
@@ -194,6 +200,15 @@ func isValidConditions(conditions map[string]map[string]set.StringSet) (err erro
err = fmt.Errorf("Unsupported condition key '%s', please validate your policy document", conditionType)
return err
}
compatibleActions := conditionKeyActionMap[key]
if !compatibleActions.IsEmpty() &&
compatibleActions.Intersection(actions).IsEmpty() {
err = fmt.Errorf("Unsupported condition key %s for the given actions %s, "+
"please validate your policy document", key, actions)
return err
}
conditionVal, ok := conditionKeyVal[key]
if ok && !value.Intersection(conditionVal).IsEmpty() {
err = fmt.Errorf("Ambigious condition values for key '%s', please validate your policy document", key)
@@ -222,8 +237,8 @@ func resourcePrefix(resource string) string {
}
// checkBucketPolicyResources validates Resources in unmarshalled bucket policy structure.
// First valation of Resources done for given set of Actions.
// Later its validated for recursive Resources.
// - Resources are validated against the given set of Actions.
// -
func checkBucketPolicyResources(bucket string, bucketPolicy *bucketPolicy) APIErrorCode {
// Validate statements for special actions and collect resources
// for others to validate nesting.
@@ -267,7 +282,7 @@ func checkBucketPolicyResources(bucket string, bucketPolicy *bucketPolicy) APIEr
// nesting. Reject such rules.
for _, otherResource := range resources {
// Common prefix reject such rules.
if strings.HasPrefix(otherResource, resource) {
if hasPrefix(otherResource, resource) {
return ErrPolicyNesting
}
}
@@ -317,7 +332,7 @@ func parseBucketPolicy(bucketPolicyReader io.Reader, policy *bucketPolicy) (err
return err
}
// Statement conditions should be valid.
if err := isValidConditions(statement.Conditions); err != nil {
if err := isValidConditions(statement.Actions, statement.Conditions); err != nil {
return err
}
}
+109 -31
View File
@@ -22,6 +22,7 @@ import (
"fmt"
"testing"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio-go/pkg/set"
)
@@ -342,6 +343,18 @@ func TestIsValidPrincipals(t *testing.T) {
}
}
// getEmptyConditionKeyMap - returns a function that generates a
// condition key map for a given key.
func getEmptyConditionKeyMap(conditionKey string) func() map[string]map[string]set.StringSet {
emptyConditonGenerator := func() map[string]map[string]set.StringSet {
emptyMap := make(map[string]set.StringSet)
conditions := make(map[string]map[string]set.StringSet)
conditions[conditionKey] = emptyMap
return conditions
}
return emptyConditonGenerator
}
// Tests validate policyStatement condition validator.
func TestIsValidConditions(t *testing.T) {
// returns empty conditions map.
@@ -350,22 +363,17 @@ func TestIsValidConditions(t *testing.T) {
}
// returns map with the "StringEquals" set to empty map.
setEmptyStringEquals := func() map[string]map[string]set.StringSet {
emptyMap := make(map[string]set.StringSet)
conditions := make(map[string]map[string]set.StringSet)
conditions["StringEquals"] = emptyMap
return conditions
}
setEmptyStringEquals := getEmptyConditionKeyMap("StringEquals")
// returns map with the "StringNotEquals" set to empty map.
setEmptyStringNotEquals := func() map[string]map[string]set.StringSet {
emptyMap := make(map[string]set.StringSet)
conditions := make(map[string]map[string]set.StringSet)
conditions["StringNotEquals"] = emptyMap
return conditions
setEmptyStringNotEquals := getEmptyConditionKeyMap("StringNotEquals")
// returns map with the "StringLike" set to empty map.
setEmptyStringLike := getEmptyConditionKeyMap("StringLike")
// returns map with the "StringNotLike" set to empty map.
setEmptyStringNotLike := getEmptyConditionKeyMap("StringNotLike")
}
// Generate conditions.
generateConditions := func(key1, key2, value string) map[string]map[string]set.StringSet {
innerMap := make(map[string]set.StringSet)
@@ -377,11 +385,11 @@ func TestIsValidConditions(t *testing.T) {
// generate ambigious conditions.
generateAmbigiousConditions := func() map[string]map[string]set.StringSet {
innerMap := make(map[string]set.StringSet)
innerMap["s3:prefix"] = set.CreateStringSet("Asia/")
prefixMap := make(map[string]set.StringSet)
prefixMap["s3:prefix"] = set.CreateStringSet("Asia/")
conditions := make(map[string]map[string]set.StringSet)
conditions["StringEquals"] = innerMap
conditions["StringNotEquals"] = innerMap
conditions["StringEquals"] = prefixMap
conditions["StringNotEquals"] = prefixMap
return conditions
}
@@ -417,13 +425,20 @@ func TestIsValidConditions(t *testing.T) {
setEmptyConditions(),
setEmptyStringEquals(),
setEmptyStringNotEquals(),
setEmptyStringLike(),
setEmptyStringNotLike(),
generateConditions("StringEquals", "s3:prefix", "Asia/"),
generateConditions("StringEquals", "s3:max-keys", "100"),
generateConditions("StringNotEquals", "s3:prefix", "Asia/"),
generateConditions("StringNotEquals", "s3:max-keys", "100"),
}
getObjectActionSet := set.CreateStringSet("s3:GetObject")
roBucketActionSet := set.CreateStringSet(readOnlyBucketActions...)
maxKeysConditionErr := fmt.Errorf("Unsupported condition key %s for the given actions %s, "+
"please validate your policy document", "s3:max-keys", getObjectActionSet)
testCases := []struct {
inputActions set.StringSet
inputCondition map[string]map[string]set.StringSet
// expected result.
expectedErr error
@@ -433,42 +448,44 @@ func TestIsValidConditions(t *testing.T) {
// Malformed conditions.
// Test case - 1.
// "StringValues" is an invalid type.
{testConditions[0], fmt.Errorf("Unsupported condition type 'StringValues', " +
{roBucketActionSet, testConditions[0], fmt.Errorf("Unsupported condition type 'StringValues', " +
"please validate your policy document"), false},
// Test case - 2.
// "s3:Object" is an invalid key.
{testConditions[1], fmt.Errorf("Unsupported condition key " +
{roBucketActionSet, testConditions[1], fmt.Errorf("Unsupported condition key " +
"'StringEquals', please validate your policy document"), false},
// Test case - 3.
// Test case with Ambigious conditions set.
{testConditions[2], fmt.Errorf("Ambigious condition values for key 's3:prefix', " +
{roBucketActionSet, testConditions[2], fmt.Errorf("Ambigious condition values for key 's3:prefix', " +
"please validate your policy document"), false},
// Test case - 4.
// Test case with valid and invalid condition types.
{testConditions[3], fmt.Errorf("Unsupported condition type 'InvalidType', " +
{roBucketActionSet, testConditions[3], fmt.Errorf("Unsupported condition type 'InvalidType', " +
"please validate your policy document"), false},
// Test case - 5.
// Test case with valid and invalid condition keys.
{testConditions[4], fmt.Errorf("Unsupported condition key 'StringEquals', " +
{roBucketActionSet, testConditions[4], fmt.Errorf("Unsupported condition key 'StringEquals', " +
"please validate your policy document"), false},
// Test cases with valid conditions.
// Test case - 6.
{testConditions[5], nil, true},
{roBucketActionSet, testConditions[5], nil, true},
// Test case - 7.
{testConditions[6], nil, true},
{roBucketActionSet, testConditions[6], nil, true},
// Test case - 8.
{testConditions[7], nil, true},
{roBucketActionSet, testConditions[7], nil, true},
// Test case - 9.
{testConditions[8], nil, true},
{roBucketActionSet, testConditions[8], nil, true},
// Test case - 10.
{testConditions[9], nil, true},
{roBucketActionSet, testConditions[9], nil, true},
// Test case - 11.
{testConditions[10], nil, true},
// Test case 10.
{testConditions[11], nil, true},
{roBucketActionSet, testConditions[10], nil, true},
// Test case - 12.
{roBucketActionSet, testConditions[11], nil, true},
// Test case - 13.
{getObjectActionSet, testConditions[11], maxKeysConditionErr, false},
}
for i, testCase := range testCases {
actualErr := isValidConditions(testCase.inputCondition)
actualErr := isValidConditions(testCase.inputActions, testCase.inputCondition)
if actualErr != nil && testCase.shouldPass {
t.Errorf("Test %d: Expected to pass, but failed with: <ERROR> %s", i+1, actualErr.Error())
}
@@ -709,3 +726,64 @@ func TestParseBucketPolicy(t *testing.T) {
}
}
}
func TestAWSRefererCondition(t *testing.T) {
resource := set.CreateStringSet([]string{
fmt.Sprintf("%s%s", bucketARNPrefix, "minio-bucket"+"/"+"Asia"+"*"),
}...)
conditionsKeyMap := make(policy.ConditionKeyMap)
conditionsKeyMap.Add("aws:Referer",
set.CreateStringSet("www.example.com",
"http://www.example.com"))
requestConditionKeyMap := make(map[string]set.StringSet)
requestConditionKeyMap["referer"] = set.CreateStringSet("www.example.com")
testCases := []struct {
effect string
conditionKey string
match bool
}{
{
effect: "Allow",
conditionKey: "StringLike",
match: true,
},
{
effect: "Allow",
conditionKey: "StringNotLike",
match: false,
},
{
effect: "Deny",
conditionKey: "StringLike",
match: true,
},
{
effect: "Deny",
conditionKey: "StringNotLike",
match: false,
},
}
for i, test := range testCases {
conditions := make(map[string]map[string]set.StringSet)
conditions[test.conditionKey] = conditionsKeyMap
allowStatement := policyStatement{
Sid: "Testing AWS referer condition",
Effect: test.effect,
Principal: map[string]interface{}{
"AWS": "*",
},
Resources: resource,
Conditions: conditions,
}
if result := bucketPolicyConditionMatch(requestConditionKeyMap, allowStatement); result != test.match {
t.Errorf("Test %d - Expected conditons to evaluate to %v but got %v",
i+1, test.match, result)
}
}
}
+65 -123
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -19,148 +19,90 @@ package cmd
import (
"crypto/x509"
"encoding/pem"
"errors"
"fmt"
"io/ioutil"
"os"
"path/filepath"
)
// createCertsPath create certs path.
func createCertsPath() error {
certsPath, err := getCertsPath()
if err != nil {
return err
}
if err := os.MkdirAll(certsPath, 0700); err != nil {
return err
}
rootCAsPath := filepath.Join(certsPath, globalMinioCertsCADir)
return os.MkdirAll(rootCAsPath, 0700)
}
// getCertsPath get certs path.
func getCertsPath() (string, error) {
var certsPath string
configDir, err := getConfigPath()
if err != nil {
return "", err
}
certsPath = filepath.Join(configDir, globalMinioCertsDir)
return certsPath, nil
}
// mustGetCertsPath must get certs path.
func mustGetCertsPath() string {
certsPath, err := getCertsPath()
fatalIf(err, "Failed to get certificate path.")
return certsPath
}
// mustGetCertFile must get cert file.
func mustGetCertFile() string {
return filepath.Join(mustGetCertsPath(), globalMinioCertFile)
}
// mustGetKeyFile must get key file.
func mustGetKeyFile() string {
return filepath.Join(mustGetCertsPath(), globalMinioKeyFile)
}
// mustGetCAFiles must get the list of the CA certificates stored in minio config dir
func mustGetCAFiles() (caCerts []string) {
CAsDir := filepath.Join(mustGetCertsPath(), globalMinioCertsCADir)
caFiles, _ := ioutil.ReadDir(CAsDir)
for _, cert := range caFiles {
caCerts = append(caCerts, filepath.Join(CAsDir, cert.Name()))
}
return
}
// mustGetSystemCertPool returns empty cert pool in case of error (windows)
func mustGetSystemCertPool() *x509.CertPool {
pool, err := x509.SystemCertPool()
if err != nil {
return x509.NewCertPool()
}
return pool
}
// isCertFileExists verifies if cert file exists, returns true if
// found, false otherwise.
func isCertFileExists() bool {
st, e := os.Stat(filepath.Join(mustGetCertsPath(), globalMinioCertFile))
// If file exists and is regular return true.
if e == nil && st.Mode().IsRegular() {
return true
}
return false
}
// isKeyFileExists verifies if key file exists, returns true if found,
// false otherwise.
func isKeyFileExists() bool {
st, e := os.Stat(filepath.Join(mustGetCertsPath(), globalMinioKeyFile))
// If file exists and is regular return true.
if e == nil && st.Mode().IsRegular() {
return true
}
return false
}
// isSSL - returns true with both cert and key exists.
func isSSL() bool {
return isCertFileExists() && isKeyFileExists()
return isFile(getPublicCertFile()) && isFile(getPrivateKeyFile())
}
// Reads certificated file and returns a list of parsed certificates.
func parsePublicCertFile(certFile string) (certs []*x509.Certificate, err error) {
var bytes []byte
if bytes, err = ioutil.ReadFile(certFile); err != nil {
return certs, err
}
// Parse all certs in the chain.
var block *pem.Block
var cert *x509.Certificate
current := bytes
for len(current) > 0 {
if block, current = pem.Decode(current); block == nil {
err = fmt.Errorf("Could not read PEM block from file %s", certFile)
return certs, err
}
if cert, err = x509.ParseCertificate(block.Bytes); err != nil {
return certs, err
}
certs = append(certs, cert)
}
if len(certs) == 0 {
err = fmt.Errorf("Empty public certificate file %s", certFile)
}
return certs, err
}
// Reads certificate file and returns a list of parsed certificates.
func readCertificateChain() ([]*x509.Certificate, error) {
bytes, err := ioutil.ReadFile(mustGetCertFile())
return parsePublicCertFile(getPublicCertFile())
}
func getRootCAs(certsCAsDir string) (*x509.CertPool, error) {
// Get all CA file names.
var caFiles []string
fis, err := ioutil.ReadDir(certsCAsDir)
if err != nil {
return nil, err
}
// Proceed to parse the certificates.
return parseCertificateChain(bytes)
}
// Parses certificate chain, returns a list of parsed certificates.
func parseCertificateChain(bytes []byte) ([]*x509.Certificate, error) {
var certs []*x509.Certificate
var block *pem.Block
current := bytes
// Parse all certs in the chain.
for len(current) > 0 {
block, current = pem.Decode(current)
if block == nil {
return nil, errors.New("Could not PEM block")
}
// Parse the decoded certificate.
cert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return nil, err
}
certs = append(certs, cert)
for _, fi := range fis {
caFiles = append(caFiles, filepath.Join(certsCAsDir, fi.Name()))
}
return certs, nil
}
// loadRootCAs fetches CA files provided in minio config and adds them to globalRootCAs
// Currently under Windows, there is no way to load system + user CAs at the same time
func loadRootCAs() {
caFiles := mustGetCAFiles()
if len(caFiles) == 0 {
return
return nil, nil
}
// Get system cert pool, and empty cert pool under Windows because it is not supported
globalRootCAs = mustGetSystemCertPool()
rootCAs, err := x509.SystemCertPool()
if err != nil {
// In some systems like Windows, system cert pool is not supported.
// Hence we create a new cert pool.
rootCAs = x509.NewCertPool()
}
// Load custom root CAs for client requests
for _, caFile := range caFiles {
caCert, err := ioutil.ReadFile(caFile)
if err != nil {
fatalIf(err, "Unable to load a CA file")
return rootCAs, err
}
globalRootCAs.AppendCertsFromPEM(caCert)
rootCAs.AppendCertsFromPEM(caCert)
}
return rootCAs, nil
}
// loadRootCAs fetches CA files provided in minio config and adds them to globalRootCAs
// Currently under Windows, there is no way to load system + user CAs at the same time
func loadRootCAs() (err error) {
globalRootCAs, err = getRootCAs(getCADir())
return err
}
+203 -55
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,52 +17,86 @@
package cmd
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strings"
"runtime"
"testing"
)
// Make sure we have a valid certs path.
func TestGetCertsPath(t *testing.T) {
path, err := getCertsPath()
func createTempFile(prefix, content string) (tempFile string, err error) {
var tmpfile *os.File
if tmpfile, err = ioutil.TempFile("", prefix); err != nil {
return tempFile, err
}
if _, err = tmpfile.Write([]byte(content)); err != nil {
return tempFile, err
}
if err = tmpfile.Close(); err != nil {
return tempFile, err
}
tempFile = tmpfile.Name()
return tempFile, err
}
func TestParsePublicCertFile(t *testing.T) {
tempFile1, err := createTempFile("public-cert-file", "")
if err != nil {
t.Error(err)
}
if path == "" {
t.Errorf("expected path to not be an empty string, got: '%s'", path)
}
// Ensure it contains some sort of path separator.
if !strings.ContainsRune(path, os.PathSeparator) {
t.Errorf("expected path to contain file separator")
}
// It should also be an absolute path.
if !filepath.IsAbs(path) {
t.Errorf("expected path to be an absolute path")
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile1)
// This will error if something goes wrong, so just call it.
mustGetCertsPath()
}
// Ensure that the certificate and key file getters contain their respective
// file name and endings.
func TestGetFiles(t *testing.T) {
file := mustGetCertFile()
if !strings.Contains(file, globalMinioCertFile) {
t.Errorf("CertFile does not contain %s", globalMinioCertFile)
tempFile2, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBA-some-junk-Q4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile2)
file = mustGetKeyFile()
if !strings.Contains(file, globalMinioKeyFile) {
t.Errorf("KeyFile does not contain %s", globalMinioKeyFile)
tempFile3, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBAabababababaQ4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Unable to create temporary file. %v", err)
}
}
defer os.Remove(tempFile3)
// Parses .crt file contents
func TestParseCertificateChain(t *testing.T) {
// given
cert := `-----BEGIN CERTIFICATE-----
tempFile4, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
@@ -77,35 +111,149 @@ JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`
// when
certs, err := parseCertificateChain([]byte(cert))
// then
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Could not parse certificate: %s", err)
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile4)
tempFile5, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBAoTBU1pbmlvMQ4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBAoTBU1pbmlvMQ4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile5)
nonexistentErr := fmt.Errorf("open nonexistent-file: no such file or directory")
if runtime.GOOS == "windows" {
// Below concatenation is done to get rid of goline error
// "error strings should not be capitalized or end with punctuation or a newline"
nonexistentErr = fmt.Errorf("open nonexistent-file:" + " The system cannot find the file specified.")
}
if len(certs) != 1 {
t.Fatalf("Expected number of certificates in chain was 1, actual: %d", len(certs))
testCases := []struct {
certFile string
expectedResultLen int
expectedErr error
}{
{"nonexistent-file", 0, nonexistentErr},
{tempFile1, 0, fmt.Errorf("Empty public certificate file %s", tempFile1)},
{tempFile2, 0, fmt.Errorf("Could not read PEM block from file %s", tempFile2)},
{tempFile3, 0, fmt.Errorf("asn1: structure error: sequence tag mismatch")},
{tempFile4, 1, nil},
{tempFile5, 2, nil},
}
if certs[0].Subject.CommonName != "Minio" {
t.Fatalf("Expected Subject.CommonName was Minio, actual: %s", certs[0].Subject.CommonName)
for _, testCase := range testCases {
certs, err := parsePublicCertFile(testCase.certFile)
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
} else if err == nil {
t.Fatalf("error: expected = %v, got = <nil>", testCase.expectedErr)
} else if testCase.expectedErr.Error() != err.Error() {
t.Fatalf("error: expected = %v, got = %v", testCase.expectedErr, err)
}
if len(certs) != testCase.expectedResultLen {
t.Fatalf("certs: expected = %v, got = %v", testCase.expectedResultLen, len(certs))
}
}
}
// Parses invalid .crt file contents and returns error
func TestParseInvalidCertificateChain(t *testing.T) {
// given
cert := `This is now valid certificate`
func TestGetRootCAs(t *testing.T) {
emptydir, err := ioutil.TempDir("", "test-get-root-cas")
if err != nil {
t.Fatalf("Unable create temp directory. %v", emptydir)
}
defer os.RemoveAll(emptydir)
// when
_, err := parseCertificateChain([]byte(cert))
dir1, err := ioutil.TempDir("", "test-get-root-cas")
if err != nil {
t.Fatalf("Unable create temp directory. %v", dir1)
}
defer os.RemoveAll(dir1)
if err = os.Mkdir(filepath.Join(dir1, "empty-dir"), 0755); err != nil {
t.Fatalf("Unable create empty dir. %v", err)
}
// then
if err == nil {
t.Fatalf("Expected error but none occurred")
dir2, err := ioutil.TempDir("", "test-get-root-cas")
if err != nil {
t.Fatalf("Unable create temp directory. %v", dir2)
}
defer os.RemoveAll(dir2)
if err = ioutil.WriteFile(filepath.Join(dir2, "empty-file"), []byte{}, 0644); err != nil {
t.Fatalf("Unable create test file. %v", err)
}
nonexistentErr := fmt.Errorf("open nonexistent-dir: no such file or directory")
if runtime.GOOS == "windows" {
// Below concatenation is done to get rid of goline error
// "error strings should not be capitalized or end with punctuation or a newline"
nonexistentErr = fmt.Errorf("open nonexistent-dir:" + " The system cannot find the file specified.")
}
err1 := fmt.Errorf("read %s: is a directory", filepath.Join(dir1, "empty-dir"))
if runtime.GOOS == "windows" {
// Below concatenation is done to get rid of goline error
// "error strings should not be capitalized or end with punctuation or a newline"
err1 = fmt.Errorf("read %s:"+" The handle is invalid.", filepath.Join(dir1, "empty-dir"))
}
testCases := []struct {
certCAsDir string
expectedErr error
}{
{"nonexistent-dir", nonexistentErr},
{dir1, err1},
{emptydir, nil},
{dir2, nil},
}
for _, testCase := range testCases {
_, err := getRootCAs(testCase.certCAsDir)
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
} else if err == nil {
t.Fatalf("error: expected = %v, got = <nil>", testCase.expectedErr)
} else if testCase.expectedErr.Error() != err.Error() {
t.Fatalf("error: expected = %v, got = %v", testCase.expectedErr, err)
}
}
}
+5 -17
View File
@@ -22,25 +22,13 @@ import (
"syscall"
)
// Make sure that none of the other processes are listening on the
// specified port on any of the interfaces.
//
// On linux if a process is listening on 127.0.0.1:9000 then Listen()
// on ":9000" fails with the error "port already in use".
// However on Mac OSX Listen() on ":9000" falls back to the IPv6 address.
// This causes confusion on Mac OSX that minio server is not reachable
// on 127.0.0.1 even though minio server is running. So before we start
// the minio server we make sure that the port is free on each tcp network.
//
// Port is string on purpose here.
// https://github.com/golang/go/issues/16142#issuecomment-245912773
//
// "Keep in mind that ports in Go are strings: https://play.golang.org/p/zk2WEri_E9"
// - @bradfitz
func checkPortAvailability(portStr string) error {
// checkPortAvailability - check if given port is already in use.
// Note: The check method tries to listen on given port and closes it.
// It is possible to have a disconnected client in this tiny window of time.
func checkPortAvailability(port string) error {
network := [3]string{"tcp", "tcp4", "tcp6"}
for _, n := range network {
l, err := net.Listen(n, net.JoinHostPort("", portStr))
l, err := net.Listen(n, net.JoinHostPort("", port))
if err != nil {
if isAddrInUse(err) {
// Return error if another process is listening on the
+139
View File
@@ -0,0 +1,139 @@
/*
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"path/filepath"
"sync"
homedir "github.com/minio/go-homedir"
"github.com/minio/mc/pkg/console"
)
const (
// Default minio configuration directory where below configuration files/directories are stored.
defaultMinioConfigDir = ".minio"
// Minio configuration file.
minioConfigFile = "config.json"
// Directory contains below files/directories for HTTPS configuration.
certsDir = "certs"
// Directory contains all CA certificates other than system defaults for HTTPS.
certsCADir = "CAs"
// Public certificate file for HTTPS.
publicCertFile = "public.crt"
// Private key file for HTTPS.
privateKeyFile = "private.key"
)
// ConfigDir - configuration directory with locking.
type ConfigDir struct {
sync.Mutex
dir string
}
// Set - saves given directory as configuration directory.
func (config *ConfigDir) Set(dir string) {
config.Lock()
defer config.Unlock()
config.dir = dir
}
// Get - returns current configuration directory.
func (config *ConfigDir) Get() string {
config.Lock()
defer config.Unlock()
return config.dir
}
func (config *ConfigDir) getCertsDir() string {
return filepath.Join(config.Get(), certsDir)
}
// GetCADir - returns certificate CA directory.
func (config *ConfigDir) GetCADir() string {
return filepath.Join(config.getCertsDir(), certsCADir)
}
// Create - creates configuration directory tree.
func (config *ConfigDir) Create() error {
return mkdirAll(config.GetCADir(), 0700)
}
// GetMinioConfigFile - returns absolute path of config.json file.
func (config *ConfigDir) GetMinioConfigFile() string {
return filepath.Join(config.Get(), minioConfigFile)
}
// GetPublicCertFile - returns absolute path of public.crt file.
func (config *ConfigDir) GetPublicCertFile() string {
return filepath.Join(config.getCertsDir(), publicCertFile)
}
// GetPrivateKeyFile - returns absolute path of private.key file.
func (config *ConfigDir) GetPrivateKeyFile() string {
return filepath.Join(config.getCertsDir(), privateKeyFile)
}
func mustGetDefaultConfigDir() string {
homeDir, err := homedir.Dir()
if err != nil {
console.Fatalln("Unable to get home directory.", err)
}
return filepath.Join(homeDir, defaultMinioConfigDir)
}
var configDir = &ConfigDir{dir: mustGetDefaultConfigDir()}
func setConfigDir(dir string) {
configDir.Set(dir)
}
func getConfigDir() string {
return configDir.Get()
}
func getCADir() string {
return configDir.GetCADir()
}
func createConfigDir() error {
return configDir.Create()
}
func getConfigFile() string {
return configDir.GetMinioConfigFile()
}
func getPublicCertFile() string {
return configDir.GetPublicCertFile()
}
func getPrivateKeyFile() string {
return configDir.GetPrivateKeyFile()
}
func isConfigFileExists() bool {
return isFile(getConfigFile())
}
+128 -59
View File
@@ -74,6 +74,10 @@ func migrateConfig() error {
if err := migrateV12ToV13(); err != nil {
return err
}
// Migration version '13' to '14'.
if err := migrateV13ToV14(); err != nil {
return err
}
return nil
}
@@ -90,15 +94,10 @@ func purgeV1() error {
}
if cv1.Version == "1" {
console.Println("Removed unsupported config version 1.")
/// Purge old fsUsers.json file
configPath, err := getConfigPath()
if err != nil {
return fmt.Errorf("Unable to retrieve config path. %v", err)
}
configFile := filepath.Join(configPath, "fsUsers.json")
// Purge old fsUsers.json file
configFile := filepath.Join(getConfigDir(), "fsUsers.json")
removeAll(configFile)
console.Println("Removed unsupported config version 1.")
return nil
}
return fmt.Errorf("Failed to migrate unrecognized config version " + cv1.Version + ".")
@@ -117,13 +116,16 @@ func migrateV2ToV3() error {
if cv2.Version != "2" {
return nil
}
cred, err := createCredential(cv2.Credentials.AccessKey, cv2.Credentials.SecretKey)
if err != nil {
return fmt.Errorf("Invalid credential in V2 configuration file. %v", err)
}
srvConfig := &configV3{}
srvConfig.Version = "3"
srvConfig.Addr = ":9000"
srvConfig.Credential = credential{
AccessKey: cv2.Credentials.AccessKey,
SecretKey: cv2.Credentials.SecretKey,
}
srvConfig.Credential = cred
srvConfig.Region = cv2.Credentials.Region
if srvConfig.Region == "" {
// Region needs to be set for AWS Signature V4.
@@ -154,12 +156,7 @@ func migrateV2ToV3() error {
return fmt.Errorf("Unable to initialize config. %v", err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
// Migrate the config.
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf("Failed to migrate config from "+cv2.Version+" to "+srvConfig.Version+" failed. %v", err)
@@ -201,11 +198,8 @@ func migrateV3ToV4() error {
if err != nil {
return fmt.Errorf("Unable to initialize the quick config. %v", err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf("Failed to migrate config from "+cv3.Version+" to "+srvConfig.Version+" failed. %v", err)
@@ -250,11 +244,8 @@ func migrateV4ToV5() error {
if err != nil {
return fmt.Errorf("Unable to initialize the quick config. %v", err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf("Failed to migrate config from "+cv4.Version+" to "+srvConfig.Version+" failed. %v", err)
@@ -326,11 +317,8 @@ func migrateV5ToV6() error {
if err != nil {
return fmt.Errorf("Unable to initialize the quick config. %v", err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf("Failed to migrate config from "+cv5.Version+" to "+srvConfig.Version+" failed. %v", err)
@@ -390,11 +378,8 @@ func migrateV6ToV7() error {
if err != nil {
return fmt.Errorf("Unable to initialize the quick config. %v", err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf("Failed to migrate config from "+cv6.Version+" to "+srvConfig.Version+" failed. %v", err)
@@ -461,11 +446,8 @@ func migrateV7ToV8() error {
if err != nil {
return fmt.Errorf("Unable to initialize the quick config. %v", err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf("Failed to migrate config from "+cv7.Version+" to "+srvConfig.Version+" failed. %v", err)
@@ -540,11 +522,8 @@ func migrateV8ToV9() error {
return fmt.Errorf("Unable to initialize the quick config. %v",
err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf(
@@ -625,11 +604,8 @@ func migrateV9ToV10() error {
return fmt.Errorf("Unable to initialize the quick config. %v",
err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf(
@@ -713,11 +689,8 @@ func migrateV10ToV11() error {
return fmt.Errorf("Unable to initialize the quick config. %v",
err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf(
@@ -819,11 +792,8 @@ func migrateV11ToV12() error {
return fmt.Errorf("Unable to initialize the quick config. %v",
err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf(
@@ -855,7 +825,10 @@ func migrateV12ToV13() error {
}
// Copy over fields from V12 into V13 config struct
srvConfig := &serverConfigV13{}
srvConfig := &serverConfigV13{
Logger: &logger{},
Notify: &notifier{},
}
srvConfig.Version = "13"
srvConfig.Credential = cv12.Credential
srvConfig.Region = cv12.Region
@@ -913,11 +886,8 @@ func migrateV12ToV13() error {
return fmt.Errorf("Unable to initialize the quick config. %v",
err)
}
configFile, err := getConfigFile()
if err != nil {
return fmt.Errorf("Unable to get config file. %v", err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf(
@@ -934,3 +904,102 @@ func migrateV12ToV13() error {
)
return nil
}
// Version '13' to '14' migration. Add support for custom webhook endpoint.
func migrateV13ToV14() error {
cv13, err := loadConfigV13()
if err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("Unable to load config version 13. %v", err)
}
if cv13.Version != "13" {
return nil
}
// Copy over fields from V13 into V14 config struct
srvConfig := &serverConfigV14{
Logger: &logger{},
Notify: &notifier{},
}
srvConfig.Version = "14"
srvConfig.Credential = cv13.Credential
srvConfig.Region = cv13.Region
if srvConfig.Region == "" {
// Region needs to be set for AWS Signature Version 4.
srvConfig.Region = globalMinioDefaultRegion
}
srvConfig.Logger.Console = cv13.Logger.Console
srvConfig.Logger.File = cv13.Logger.File
// check and set notifiers config
if len(cv13.Notify.AMQP) == 0 {
srvConfig.Notify.AMQP = make(map[string]amqpNotify)
srvConfig.Notify.AMQP["1"] = amqpNotify{}
} else {
srvConfig.Notify.AMQP = cv13.Notify.AMQP
}
if len(cv13.Notify.ElasticSearch) == 0 {
srvConfig.Notify.ElasticSearch = make(map[string]elasticSearchNotify)
srvConfig.Notify.ElasticSearch["1"] = elasticSearchNotify{}
} else {
srvConfig.Notify.ElasticSearch = cv13.Notify.ElasticSearch
}
if len(cv13.Notify.Redis) == 0 {
srvConfig.Notify.Redis = make(map[string]redisNotify)
srvConfig.Notify.Redis["1"] = redisNotify{}
} else {
srvConfig.Notify.Redis = cv13.Notify.Redis
}
if len(cv13.Notify.PostgreSQL) == 0 {
srvConfig.Notify.PostgreSQL = make(map[string]postgreSQLNotify)
srvConfig.Notify.PostgreSQL["1"] = postgreSQLNotify{}
} else {
srvConfig.Notify.PostgreSQL = cv13.Notify.PostgreSQL
}
if len(cv13.Notify.Kafka) == 0 {
srvConfig.Notify.Kafka = make(map[string]kafkaNotify)
srvConfig.Notify.Kafka["1"] = kafkaNotify{}
} else {
srvConfig.Notify.Kafka = cv13.Notify.Kafka
}
if len(cv13.Notify.NATS) == 0 {
srvConfig.Notify.NATS = make(map[string]natsNotify)
srvConfig.Notify.NATS["1"] = natsNotify{}
} else {
srvConfig.Notify.NATS = cv13.Notify.NATS
}
if len(cv13.Notify.Webhook) == 0 {
srvConfig.Notify.Webhook = make(map[string]webhookNotify)
srvConfig.Notify.Webhook["1"] = webhookNotify{}
} else {
srvConfig.Notify.Webhook = cv13.Notify.Webhook
}
// Set the new browser parameter to true by default
srvConfig.Browser = "on"
qc, err := quick.New(srvConfig)
if err != nil {
return fmt.Errorf("Unable to initialize the quick config. %v",
err)
}
configFile := getConfigFile()
err = qc.Save(configFile)
if err != nil {
return fmt.Errorf(
"Failed to migrate config from "+
cv13.Version+" to "+srvConfig.Version+
" failed. %v", err,
)
}
console.Println(
"Migration from version " +
cv13.Version + " to " + srvConfig.Version +
" completed successfully.",
)
return nil
}
+17 -16
View File
@@ -31,7 +31,7 @@ func TestServerConfigMigrateV1(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
setConfigDir(rootPath)
// Create a V1 config json file and store it
configJSON := "{ \"version\":\"1\", \"accessKeyId\":\"abcde\", \"secretAccessKey\":\"abcdefgh\"}"
@@ -50,7 +50,7 @@ func TestServerConfigMigrateV1(t *testing.T) {
}
// Initialize server config and check again if everything is fine
if _, err := initConfig(); err != nil {
if err := loadConfig(envParams{}); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
}
@@ -65,8 +65,8 @@ func TestServerConfigMigrateInexistentConfig(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
configPath := rootPath + "/" + globalMinioConfigFile
setConfigDir(rootPath)
configPath := rootPath + "/" + minioConfigFile
// Remove config file
if err := os.Remove(configPath); err != nil {
@@ -106,10 +106,13 @@ func TestServerConfigMigrateInexistentConfig(t *testing.T) {
if err := migrateV12ToV13(); err != nil {
t.Fatal("migrate v12 to v13 should succeed when no config file is found")
}
if err := migrateV13ToV14(); err != nil {
t.Fatal("migrate v13 to v14 should succeed when no config file is found")
}
}
// Test if a config migration from v2 to v12 is successfully done
func TestServerConfigMigrateV2toV12(t *testing.T) {
func TestServerConfigMigrateV2toV14(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("Init Test config failed")
@@ -117,8 +120,8 @@ func TestServerConfigMigrateV2toV12(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
configPath := rootPath + "/" + globalMinioConfigFile
setConfigDir(rootPath)
configPath := rootPath + "/" + minioConfigFile
// Create a corrupted config file
if err := ioutil.WriteFile(configPath, []byte("{ \"version\":\"2\","), 0644); err != nil {
@@ -143,12 +146,12 @@ func TestServerConfigMigrateV2toV12(t *testing.T) {
}
// Initialize server config and check again if everything is fine
if _, err := initConfig(); err != nil {
if err := loadConfig(envParams{}); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
// Check the version number in the upgraded config file
expectedVersion := globalMinioConfigVersion
expectedVersion := v14
if serverConfig.Version != expectedVersion {
t.Fatalf("Expect version "+expectedVersion+", found: %v", serverConfig.Version)
}
@@ -160,11 +163,6 @@ func TestServerConfigMigrateV2toV12(t *testing.T) {
if serverConfig.Credential.SecretKey != secretKey {
t.Fatalf("Secret key lost during migration, expected: %v, found: %v", secretKey, serverConfig.Credential.SecretKey)
}
// Initialize server config and check again if everything is fine
if _, err := initConfig(); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
}
// Test if all migrate code returns error with corrupted config files
@@ -176,8 +174,8 @@ func TestServerConfigMigrateFaultyConfig(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
configPath := rootPath + "/" + globalMinioConfigFile
setConfigDir(rootPath)
configPath := rootPath + "/" + minioConfigFile
// Create a corrupted config file
if err := ioutil.WriteFile(configPath, []byte("{ \"version\":\""), 0644); err != nil {
@@ -218,4 +216,7 @@ func TestServerConfigMigrateFaultyConfig(t *testing.T) {
if err := migrateV12ToV13(); err == nil {
t.Fatal("migrateConfigV12ToV13() should fail with a corrupted json")
}
if err := migrateV13ToV14(); err == nil {
t.Fatal("migrateConfigV13ToV14() should fail with a corrupted json")
}
}
+106 -182
View File
@@ -1,3 +1,19 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
@@ -8,6 +24,23 @@ import (
"github.com/minio/minio/pkg/quick"
)
func loadOldConfig(configFile string, config interface{}) (interface{}, error) {
if _, err := os.Stat(configFile); err != nil {
return nil, err
}
qc, err := quick.New(config)
if err != nil {
return nil, err
}
if err = qc.Load(configFile); err != nil {
return nil, err
}
return config, nil
}
/////////////////// Config V1 ///////////////////
type configV1 struct {
Version string `json:"version"`
@@ -17,24 +50,12 @@ type configV1 struct {
// loadConfigV1 load config
func loadConfigV1() (*configV1, error) {
configPath, err := getConfigPath()
if err != nil {
configFile := filepath.Join(getConfigDir(), "fsUsers.json")
config, err := loadOldConfig(configFile, &configV1{Version: "1"})
if config == nil {
return nil, err
}
configFile := filepath.Join(configPath, "fsUsers.json")
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV1{}
c.Version = "1"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*configV1), err
}
/////////////////// Config V2 ///////////////////
@@ -61,23 +82,12 @@ type configV2 struct {
// loadConfigV2 load config version '2'.
func loadConfigV2() (*configV2, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &configV2{Version: "2"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV2{}
c.Version = "2"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*configV2), err
}
/////////////////// Config V3 ///////////////////
@@ -135,23 +145,12 @@ type configV3 struct {
// loadConfigV3 load config version '3'.
func loadConfigV3() (*configV3, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &configV3{Version: "3"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV3{}
c.Version = "3"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*configV3), err
}
// logger type representing version '4' logger config.
@@ -186,23 +185,12 @@ type configV4 struct {
// loadConfigV4 load config version '4'.
func loadConfigV4() (*configV4, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &configV4{Version: "4"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV4{}
c.Version = "4"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*configV4), err
}
// logger type representing version '5' logger config.
@@ -264,23 +252,12 @@ type configV5 struct {
// loadConfigV5 load config version '5'.
func loadConfigV5() (*configV5, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &configV5{Version: "5"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV5{}
c.Version = "5"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*configV5), err
}
type loggerV6 struct {
@@ -306,23 +283,12 @@ type configV6 struct {
// loadConfigV6 load config version '6'.
func loadConfigV6() (*configV6, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &configV6{Version: "6"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV6{}
c.Version = "6"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*configV6), err
}
// Notifier represents collection of supported notification queues in version
@@ -367,23 +333,12 @@ type serverConfigV7 struct {
// loadConfigV7 load config version '7'.
func loadConfigV7() (*serverConfigV7, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &serverConfigV7{Version: "7"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &serverConfigV7{}
c.Version = "7"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*serverConfigV7), err
}
// serverConfigV8 server configuration version '8'. Adds NATS notifier
@@ -407,23 +362,12 @@ type serverConfigV8 struct {
// loadConfigV8 load config version '8'.
func loadConfigV8() (*serverConfigV8, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &serverConfigV8{Version: "8"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &serverConfigV8{}
c.Version = "8"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
return config.(*serverConfigV8), err
}
// serverConfigV9 server configuration version '9'. Adds PostgreSQL
@@ -446,24 +390,12 @@ type serverConfigV9 struct {
}
func loadConfigV9() (*serverConfigV9, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &serverConfigV9{Version: "9"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV9{}
srvCfg.Version = "9"
srvCfg.rwMutex = &sync.RWMutex{}
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
return config.(*serverConfigV9), err
}
// serverConfigV10 server configuration version '10' which is like
@@ -484,23 +416,12 @@ type serverConfigV10 struct {
}
func loadConfigV10() (*serverConfigV10, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &serverConfigV10{Version: "10"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV10{}
srvCfg.Version = "10"
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
return config.(*serverConfigV10), err
}
// natsNotifyV1 - structure was valid until config V 11
@@ -532,23 +453,12 @@ type serverConfigV11 struct {
}
func loadConfigV11() (*serverConfigV11, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &serverConfigV11{Version: "11"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV11{}
srvCfg.Version = "11"
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
return config.(*serverConfigV11), err
}
// serverConfigV12 server configuration version '12' which is like
@@ -568,21 +478,35 @@ type serverConfigV12 struct {
}
func loadConfigV12() (*serverConfigV12, error) {
configFile, err := getConfigFile()
if err != nil {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &serverConfigV12{Version: "12"})
if config == nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV12{}
srvCfg.Version = "12"
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
return config.(*serverConfigV12), err
}
// serverConfigV13 server configuration version '13' which is like
// version '12' except it adds support for webhook notification.
type serverConfigV13 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
// Additional error logging configuration.
Logger *logger `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
func loadConfigV13() (*serverConfigV13, error) {
configFile := getConfigFile()
config, err := loadOldConfig(configFile, &serverConfigV13{Version: "13"})
if config == nil {
return nil, err
}
return config.(*serverConfigV13), err
}
-370
View File
@@ -1,370 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"os"
"sync"
"github.com/minio/minio/pkg/quick"
)
// Read Write mutex for safe access to ServerConfig.
var serverConfigMu sync.RWMutex
// serverConfigV13 server configuration version '13' which is like
// version '12' except it adds support for webhook notification.
type serverConfigV13 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
// Additional error logging configuration.
Logger logger `json:"logger"`
// Notification queue configuration.
Notify notifier `json:"notify"`
}
// initConfig - initialize server config and indicate if we are
// creating a new file or we are just loading
func initConfig() (bool, error) {
if !isConfigFileExists() {
// Initialize server config.
srvCfg := &serverConfigV13{}
srvCfg.Version = globalMinioConfigVersion
srvCfg.Region = globalMinioDefaultRegion
srvCfg.Credential = newCredential()
// Enable console logger by default on a fresh run.
srvCfg.Logger.Console = consoleLogger{
Enable: true,
Level: "error",
}
// Make sure to initialize notification configs.
srvCfg.Notify.AMQP = make(map[string]amqpNotify)
srvCfg.Notify.AMQP["1"] = amqpNotify{}
srvCfg.Notify.ElasticSearch = make(map[string]elasticSearchNotify)
srvCfg.Notify.ElasticSearch["1"] = elasticSearchNotify{}
srvCfg.Notify.Redis = make(map[string]redisNotify)
srvCfg.Notify.Redis["1"] = redisNotify{}
srvCfg.Notify.NATS = make(map[string]natsNotify)
srvCfg.Notify.NATS["1"] = natsNotify{}
srvCfg.Notify.PostgreSQL = make(map[string]postgreSQLNotify)
srvCfg.Notify.PostgreSQL["1"] = postgreSQLNotify{}
srvCfg.Notify.Kafka = make(map[string]kafkaNotify)
srvCfg.Notify.Kafka["1"] = kafkaNotify{}
srvCfg.Notify.Webhook = make(map[string]webhookNotify)
srvCfg.Notify.Webhook["1"] = webhookNotify{}
// Create config path.
err := createConfigPath()
if err != nil {
return false, err
}
// hold the mutex lock before a new config is assigned.
// Save the new config globally.
// unlock the mutex.
serverConfigMu.Lock()
serverConfig = srvCfg
serverConfigMu.Unlock()
// Save config into file.
return true, serverConfig.Save()
}
configFile, err := getConfigFile()
if err != nil {
return false, err
}
if _, err = os.Stat(configFile); err != nil {
return false, err
}
srvCfg := &serverConfigV13{}
srvCfg.Version = globalMinioConfigVersion
qc, err := quick.New(srvCfg)
if err != nil {
return false, err
}
if err := qc.Load(configFile); err != nil {
return false, err
}
// hold the mutex lock before a new config is assigned.
serverConfigMu.Lock()
// Save the loaded config globally.
serverConfig = srvCfg
serverConfigMu.Unlock()
// Set the version properly after the unmarshalled json is loaded.
serverConfig.Version = globalMinioConfigVersion
return false, nil
}
// serverConfig server config.
var serverConfig *serverConfigV13
// GetVersion get current config version.
func (s serverConfigV13) GetVersion() string {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Version
}
/// Logger related.
func (s *serverConfigV13) SetAMQPNotifyByID(accountID string, amqpn amqpNotify) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Notify.AMQP[accountID] = amqpn
}
func (s serverConfigV13) GetAMQP() map[string]amqpNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.AMQP
}
// GetAMQPNotify get current AMQP logger.
func (s serverConfigV13) GetAMQPNotifyByID(accountID string) amqpNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.AMQP[accountID]
}
//
func (s *serverConfigV13) SetNATSNotifyByID(accountID string, natsn natsNotify) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Notify.NATS[accountID] = natsn
}
func (s serverConfigV13) GetNATS() map[string]natsNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.NATS
}
// GetNATSNotify get current NATS logger.
func (s serverConfigV13) GetNATSNotifyByID(accountID string) natsNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.NATS[accountID]
}
func (s *serverConfigV13) SetElasticSearchNotifyByID(accountID string, esNotify elasticSearchNotify) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Notify.ElasticSearch[accountID] = esNotify
}
func (s serverConfigV13) GetElasticSearch() map[string]elasticSearchNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.ElasticSearch
}
// GetElasticSearchNotify get current ElasicSearch logger.
func (s serverConfigV13) GetElasticSearchNotifyByID(accountID string) elasticSearchNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.ElasticSearch[accountID]
}
func (s *serverConfigV13) SetRedisNotifyByID(accountID string, rNotify redisNotify) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Notify.Redis[accountID] = rNotify
}
func (s serverConfigV13) GetRedis() map[string]redisNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.Redis
}
func (s serverConfigV13) GetWebhook() map[string]webhookNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.Webhook
}
// GetWebhookNotifyByID get current Webhook logger.
func (s serverConfigV13) GetWebhookNotifyByID(accountID string) webhookNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.Webhook[accountID]
}
func (s *serverConfigV13) SetWebhookNotifyByID(accountID string, pgn webhookNotify) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Notify.Webhook[accountID] = pgn
}
// GetRedisNotify get current Redis logger.
func (s serverConfigV13) GetRedisNotifyByID(accountID string) redisNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.Redis[accountID]
}
func (s *serverConfigV13) SetPostgreSQLNotifyByID(accountID string, pgn postgreSQLNotify) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Notify.PostgreSQL[accountID] = pgn
}
func (s serverConfigV13) GetPostgreSQL() map[string]postgreSQLNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.PostgreSQL
}
func (s serverConfigV13) GetPostgreSQLNotifyByID(accountID string) postgreSQLNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.PostgreSQL[accountID]
}
// Kafka related functions
func (s *serverConfigV13) SetKafkaNotifyByID(accountID string, kn kafkaNotify) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Notify.Kafka[accountID] = kn
}
func (s serverConfigV13) GetKafka() map[string]kafkaNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.Kafka
}
func (s serverConfigV13) GetKafkaNotifyByID(accountID string) kafkaNotify {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Notify.Kafka[accountID]
}
// SetFileLogger set new file logger.
func (s *serverConfigV13) SetFileLogger(flogger fileLogger) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Logger.File = flogger
}
// GetFileLogger get current file logger.
func (s serverConfigV13) GetFileLogger() fileLogger {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Logger.File
}
// SetConsoleLogger set new console logger.
func (s *serverConfigV13) SetConsoleLogger(clogger consoleLogger) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Logger.Console = clogger
}
// GetConsoleLogger get current console logger.
func (s serverConfigV13) GetConsoleLogger() consoleLogger {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Logger.Console
}
// SetRegion set new region.
func (s *serverConfigV13) SetRegion(region string) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Region = region
}
// GetRegion get current region.
func (s serverConfigV13) GetRegion() string {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Region
}
// SetCredentials set new credentials.
func (s *serverConfigV13) SetCredential(creds credential) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Credential = creds
}
// GetCredentials get current credentials.
func (s serverConfigV13) GetCredential() credential {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Credential
}
// Save config.
func (s serverConfigV13) Save() error {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
// get config file.
configFile, err := getConfigFile()
if err != nil {
return err
}
// initialize quick.
qc, err := quick.New(&s)
if err != nil {
return err
}
// Save config file.
return qc.Save(configFile)
}
-112
View File
@@ -1,112 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"reflect"
"testing"
)
func TestServerConfig(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("Init Test config failed")
}
// remove the root directory after the test ends.
defer removeAll(rootPath)
if serverConfig.GetRegion() != globalMinioDefaultRegion {
t.Errorf("Expecting region `us-east-1` found %s", serverConfig.GetRegion())
}
// Set new region and verify.
serverConfig.SetRegion("us-west-1")
if serverConfig.GetRegion() != "us-west-1" {
t.Errorf("Expecting region `us-west-1` found %s", serverConfig.GetRegion())
}
// Set new amqp notification id.
serverConfig.SetAMQPNotifyByID("2", amqpNotify{})
savedNotifyCfg1 := serverConfig.GetAMQPNotifyByID("2")
if !reflect.DeepEqual(savedNotifyCfg1, amqpNotify{}) {
t.Errorf("Expecting AMQP config %#v found %#v", amqpNotify{}, savedNotifyCfg1)
}
// Set new elastic search notification id.
serverConfig.SetElasticSearchNotifyByID("2", elasticSearchNotify{})
savedNotifyCfg2 := serverConfig.GetElasticSearchNotifyByID("2")
if !reflect.DeepEqual(savedNotifyCfg2, elasticSearchNotify{}) {
t.Errorf("Expecting Elasticsearch config %#v found %#v", elasticSearchNotify{}, savedNotifyCfg2)
}
// Set new redis notification id.
serverConfig.SetRedisNotifyByID("2", redisNotify{})
savedNotifyCfg3 := serverConfig.GetRedisNotifyByID("2")
if !reflect.DeepEqual(savedNotifyCfg3, redisNotify{}) {
t.Errorf("Expecting Redis config %#v found %#v", redisNotify{}, savedNotifyCfg3)
}
// Set new kafka notification id.
serverConfig.SetKafkaNotifyByID("2", kafkaNotify{})
savedNotifyCfg4 := serverConfig.GetKafkaNotifyByID("2")
if !reflect.DeepEqual(savedNotifyCfg4, kafkaNotify{}) {
t.Errorf("Expecting Kafka config %#v found %#v", kafkaNotify{}, savedNotifyCfg4)
}
// Set new Webhook notification id.
serverConfig.SetWebhookNotifyByID("2", webhookNotify{})
savedNotifyCfg5 := serverConfig.GetWebhookNotifyByID("2")
if !reflect.DeepEqual(savedNotifyCfg5, webhookNotify{}) {
t.Errorf("Expecting Webhook config %#v found %#v", webhookNotify{}, savedNotifyCfg3)
}
// Set new console logger.
serverConfig.SetConsoleLogger(consoleLogger{
Enable: true,
})
consoleCfg := serverConfig.GetConsoleLogger()
if !reflect.DeepEqual(consoleCfg, consoleLogger{Enable: true}) {
t.Errorf("Expecting console logger config %#v found %#v", consoleLogger{Enable: true}, consoleCfg)
}
// Set new file logger.
serverConfig.SetFileLogger(fileLogger{
Enable: true,
})
fileCfg := serverConfig.GetFileLogger()
if !reflect.DeepEqual(fileCfg, fileLogger{Enable: true}) {
t.Errorf("Expecting file logger config %#v found %#v", fileLogger{Enable: true}, consoleCfg)
}
// Match version.
if serverConfig.GetVersion() != globalMinioConfigVersion {
t.Errorf("Expecting version %s found %s", serverConfig.GetVersion(), globalMinioConfigVersion)
}
// Attempt to save.
if err := serverConfig.Save(); err != nil {
t.Fatalf("Unable to save updated config file %s", err)
}
// Do this only once here.
setGlobalConfigPath(rootPath)
// Initialize server config.
if _, err := initConfig(); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
}
+370
View File
@@ -0,0 +1,370 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"errors"
"fmt"
"io/ioutil"
"os"
"strings"
"sync"
"github.com/minio/minio/pkg/quick"
"github.com/tidwall/gjson"
)
// Read Write mutex for safe access to ServerConfig.
var serverConfigMu sync.RWMutex
// Config version
var v14 = "14"
// serverConfigV14 server configuration version '14' which is like
// version '13' except it adds support of browser param.
type serverConfigV14 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
Browser string `json:"browser"`
// Additional error logging configuration.
Logger *logger `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
func newServerConfigV14() *serverConfigV14 {
srvCfg := &serverConfigV14{
Version: v14,
Region: globalMinioDefaultRegion,
Logger: &logger{},
Notify: &notifier{},
}
srvCfg.SetCredential(mustGetNewCredential())
srvCfg.SetBrowser("on")
// Enable console logger by default on a fresh run.
srvCfg.Logger.Console = consoleLogger{
Enable: true,
Level: "error",
}
// Make sure to initialize notification configs.
srvCfg.Notify.AMQP = make(map[string]amqpNotify)
srvCfg.Notify.AMQP["1"] = amqpNotify{}
srvCfg.Notify.ElasticSearch = make(map[string]elasticSearchNotify)
srvCfg.Notify.ElasticSearch["1"] = elasticSearchNotify{}
srvCfg.Notify.Redis = make(map[string]redisNotify)
srvCfg.Notify.Redis["1"] = redisNotify{}
srvCfg.Notify.NATS = make(map[string]natsNotify)
srvCfg.Notify.NATS["1"] = natsNotify{}
srvCfg.Notify.PostgreSQL = make(map[string]postgreSQLNotify)
srvCfg.Notify.PostgreSQL["1"] = postgreSQLNotify{}
srvCfg.Notify.Kafka = make(map[string]kafkaNotify)
srvCfg.Notify.Kafka["1"] = kafkaNotify{}
srvCfg.Notify.Webhook = make(map[string]webhookNotify)
srvCfg.Notify.Webhook["1"] = webhookNotify{}
return srvCfg
}
// newConfig - initialize a new server config, saves env parameters if
// found, otherwise use default parameters
func newConfig(envParams envParams) error {
// Initialize server config.
srvCfg := newServerConfigV14()
// If env is set for a fresh start, save them to config file.
if globalIsEnvCreds {
srvCfg.SetCredential(envParams.creds)
}
if globalIsEnvBrowser {
srvCfg.SetBrowser(envParams.browser)
}
// Create config path.
if err := createConfigDir(); err != nil {
return err
}
// hold the mutex lock before a new config is assigned.
// Save the new config globally.
// unlock the mutex.
serverConfigMu.Lock()
serverConfig = srvCfg
serverConfigMu.Unlock()
// Save config into file.
return serverConfig.Save()
}
// loadConfig - loads a new config from disk, overrides params from env
// if found and valid
func loadConfig(envParams envParams) error {
configFile := getConfigFile()
if _, err := os.Stat(configFile); err != nil {
return err
}
srvCfg := &serverConfigV14{}
qc, err := quick.New(srvCfg)
if err != nil {
return err
}
if err = qc.Load(configFile); err != nil {
return err
}
// If env is set override the credentials from config file.
if globalIsEnvCreds {
srvCfg.SetCredential(envParams.creds)
}
if globalIsEnvBrowser {
srvCfg.SetBrowser(envParams.browser)
}
if strings.ToLower(srvCfg.GetBrowser()) == "off" {
globalIsBrowserEnabled = false
}
// hold the mutex lock before a new config is assigned.
serverConfigMu.Lock()
// Save the loaded config globally.
serverConfig = srvCfg
serverConfigMu.Unlock()
if serverConfig.Version != v14 {
return errors.New("Unsupported config version `" + serverConfig.Version + "`.")
}
return nil
}
// doCheckDupJSONKeys recursively detects duplicate json keys
func doCheckDupJSONKeys(key, value gjson.Result) error {
// Key occurrences map of the current scope to count
// if there is any duplicated json key.
keysOcc := make(map[string]int)
// Holds the found error
var checkErr error
// Iterate over keys in the current json scope
value.ForEach(func(k, v gjson.Result) bool {
// If current key is not null, check if its
// value contains some duplicated keys.
if k.Type != gjson.Null {
keysOcc[k.String()]++
checkErr = doCheckDupJSONKeys(k, v)
}
return checkErr == nil
})
// Check found err
if checkErr != nil {
return errors.New(key.String() + " => " + checkErr.Error())
}
// Check for duplicated keys
for k, v := range keysOcc {
if v > 1 {
return errors.New(key.String() + " => `" + k + "` entry is duplicated")
}
}
return nil
}
// Check recursively if a key is duplicated in the same json scope
// e.g.:
// `{ "key" : { "key" ..` is accepted
// `{ "key" : { "subkey" : "val1", "subkey": "val2" ..` throws subkey duplicated error
func checkDupJSONKeys(json string) error {
// Parse config with gjson library
config := gjson.Parse(json)
// Create a fake rootKey since root json doesn't seem to have representation
// in gjson library.
rootKey := gjson.Result{Type: gjson.String, Str: minioConfigFile}
// Check if loaded json contains any duplicated keys
return doCheckDupJSONKeys(rootKey, config)
}
// validateConfig checks for
func validateConfig() error {
// Get file config path
configFile := getConfigFile()
srvCfg := &serverConfigV14{}
// Load config file
qc, err := quick.New(srvCfg)
if err != nil {
return err
}
if err = qc.Load(configFile); err != nil {
return err
}
// Check if config version is valid
if srvCfg.GetVersion() != v14 {
return errors.New("bad config version, expected: " + v14)
}
// Load config file json and check for duplication json keys
jsonBytes, err := ioutil.ReadFile(configFile)
if err != nil {
return err
}
if err := checkDupJSONKeys(string(jsonBytes)); err != nil {
return err
}
// Validate region field
if srvCfg.GetRegion() == "" {
return errors.New("Region config value cannot be empty")
}
// Validate browser field
if b := strings.ToLower(srvCfg.GetBrowser()); b != "on" && b != "off" {
return fmt.Errorf("Browser config value %s is invalid", b)
}
// Validate credential field
if !srvCfg.Credential.IsValid() {
return errors.New("invalid credential")
}
// Validate logger field
if err := srvCfg.Logger.Validate(); err != nil {
return err
}
// Validate notify field
if err := srvCfg.Notify.Validate(); err != nil {
return err
}
return nil
}
// serverConfig server config.
var serverConfig *serverConfigV14
// GetVersion get current config version.
func (s serverConfigV14) GetVersion() string {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Version
}
// SetRegion set new region.
func (s *serverConfigV14) SetRegion(region string) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
// Empty region means "us-east-1" by default from S3 spec.
if region == "" {
region = "us-east-1"
}
s.Region = region
}
// GetRegion get current region.
func (s serverConfigV14) GetRegion() string {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
if s.Region != "" {
return s.Region
} // region empty
// Empty region means "us-east-1" by default from S3 spec.
return "us-east-1"
}
// SetCredentials set new credentials.
func (s *serverConfigV14) SetCredential(creds credential) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
// Set updated credential.
s.Credential = creds
}
// GetCredentials get current credentials.
func (s serverConfigV14) GetCredential() credential {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Credential
}
// SetBrowser set if browser is enabled.
func (s *serverConfigV14) SetBrowser(v string) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
// Set browser param
if v == "" {
v = "on" // Browser is on by default.
}
// Set the new value.
s.Browser = v
}
// GetCredentials get current credentials.
func (s serverConfigV14) GetBrowser() string {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
if s.Browser != "" {
return s.Browser
} // empty browser.
// Empty browser means "on" by default.
return "on"
}
// Save config.
func (s serverConfigV14) Save() error {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
// get config file.
configFile := getConfigFile()
// initialize quick.
qc, err := quick.New(&s)
if err != nil {
return err
}
// Save config file.
return qc.Save(configFile)
}
+292
View File
@@ -0,0 +1,292 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"io/ioutil"
"os"
"path/filepath"
"reflect"
"testing"
"github.com/tidwall/gjson"
)
func TestServerConfig(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("Init Test config failed")
}
// remove the root directory after the test ends.
defer removeAll(rootPath)
if serverConfig.GetRegion() != globalMinioDefaultRegion {
t.Errorf("Expecting region `us-east-1` found %s", serverConfig.GetRegion())
}
// Set new region and verify.
serverConfig.SetRegion("us-west-1")
if serverConfig.GetRegion() != "us-west-1" {
t.Errorf("Expecting region `us-west-1` found %s", serverConfig.GetRegion())
}
// Set new amqp notification id.
serverConfig.Notify.SetAMQPByID("2", amqpNotify{})
savedNotifyCfg1 := serverConfig.Notify.GetAMQPByID("2")
if !reflect.DeepEqual(savedNotifyCfg1, amqpNotify{}) {
t.Errorf("Expecting AMQP config %#v found %#v", amqpNotify{}, savedNotifyCfg1)
}
// Set new elastic search notification id.
serverConfig.Notify.SetElasticSearchByID("2", elasticSearchNotify{})
savedNotifyCfg2 := serverConfig.Notify.GetElasticSearchByID("2")
if !reflect.DeepEqual(savedNotifyCfg2, elasticSearchNotify{}) {
t.Errorf("Expecting Elasticsearch config %#v found %#v", elasticSearchNotify{}, savedNotifyCfg2)
}
// Set new redis notification id.
serverConfig.Notify.SetRedisByID("2", redisNotify{})
savedNotifyCfg3 := serverConfig.Notify.GetRedisByID("2")
if !reflect.DeepEqual(savedNotifyCfg3, redisNotify{}) {
t.Errorf("Expecting Redis config %#v found %#v", redisNotify{}, savedNotifyCfg3)
}
// Set new kafka notification id.
serverConfig.Notify.SetKafkaByID("2", kafkaNotify{})
savedNotifyCfg4 := serverConfig.Notify.GetKafkaByID("2")
if !reflect.DeepEqual(savedNotifyCfg4, kafkaNotify{}) {
t.Errorf("Expecting Kafka config %#v found %#v", kafkaNotify{}, savedNotifyCfg4)
}
// Set new Webhook notification id.
serverConfig.Notify.SetWebhookByID("2", webhookNotify{})
savedNotifyCfg5 := serverConfig.Notify.GetWebhookByID("2")
if !reflect.DeepEqual(savedNotifyCfg5, webhookNotify{}) {
t.Errorf("Expecting Webhook config %#v found %#v", webhookNotify{}, savedNotifyCfg3)
}
// Set new console logger.
serverConfig.Logger.SetConsole(consoleLogger{
Enable: true,
})
consoleCfg := serverConfig.Logger.GetConsole()
if !reflect.DeepEqual(consoleCfg, consoleLogger{Enable: true}) {
t.Errorf("Expecting console logger config %#v found %#v", consoleLogger{Enable: true}, consoleCfg)
}
// Set new console logger.
serverConfig.Logger.SetConsole(consoleLogger{
Enable: false,
})
// Set new file logger.
serverConfig.Logger.SetFile(fileLogger{
Enable: true,
})
fileCfg := serverConfig.Logger.GetFile()
if !reflect.DeepEqual(fileCfg, fileLogger{Enable: true}) {
t.Errorf("Expecting file logger config %#v found %#v", fileLogger{Enable: true}, consoleCfg)
}
// Set new file logger.
serverConfig.Logger.SetFile(fileLogger{
Enable: false,
})
// Match version.
if serverConfig.GetVersion() != v14 {
t.Errorf("Expecting version %s found %s", serverConfig.GetVersion(), v14)
}
// Attempt to save.
if err := serverConfig.Save(); err != nil {
t.Fatalf("Unable to save updated config file %s", err)
}
// Do this only once here.
setConfigDir(rootPath)
// Initialize server config.
if err := loadConfig(envParams{}); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
}
func TestServerConfigWithEnvs(t *testing.T) {
os.Setenv("MINIO_BROWSER", "off")
defer os.Unsetenv("MINIO_BROWSER")
os.Setenv("MINIO_ACCESS_KEY", "minio")
defer os.Unsetenv("MINIO_ACCESS_KEY")
os.Setenv("MINIO_SECRET_KEY", "minio123")
defer os.Unsetenv("MINIO_SECRET_KEY")
defer func() {
globalIsEnvBrowser = false
globalIsEnvCreds = false
}()
// Get test root.
rootPath, err := getTestRoot()
if err != nil {
t.Error(err)
}
// Do this only once here.
setConfigDir(rootPath)
// Init config
initConfig()
// remove the root directory after the test ends.
defer removeAll(rootPath)
// Check if serverConfig has
if serverConfig.GetBrowser() != "off" {
t.Errorf("Expecting browser `off` found %s", serverConfig.GetBrowser())
}
// Check if serverConfig has
cred := serverConfig.GetCredential()
if cred.AccessKey != "minio" {
t.Errorf("Expecting access key to be `minio` found %s", cred.AccessKey)
}
if cred.SecretKey != "minio123" {
t.Errorf("Expecting access key to be `minio123` found %s", cred.SecretKey)
}
}
func TestCheckDupJSONKeys(t *testing.T) {
testCases := []struct {
json string
shouldPass bool
}{
{`{}`, true},
{`{"version" : "13"}`, true},
{`{"version" : "13", "version": "14"}`, false},
{`{"version" : "13", "credential": {"accessKey": "12345"}}`, true},
{`{"version" : "13", "credential": {"accessKey": "12345", "accessKey":"12345"}}`, false},
{`{"version" : "13", "notify": {"amqp": {"1"}, "webhook":{"3"}}}`, true},
{`{"version" : "13", "notify": {"amqp": {"1"}, "amqp":{"3"}}}`, false},
{`{"version" : "13", "notify": {"amqp": {"1":{}, "2":{}}}}`, true},
{`{"version" : "13", "notify": {"amqp": {"1":{}, "1":{}}}}`, false},
}
for i, testCase := range testCases {
err := doCheckDupJSONKeys(gjson.Result{}, gjson.Parse(testCase.json))
if testCase.shouldPass && err != nil {
t.Errorf("Test %d, should pass but it failed with err = %v", i+1, err)
}
if !testCase.shouldPass && err == nil {
t.Errorf("Test %d, should fail but it succeed.", i+1)
}
}
}
// Tests config validator..
func TestValidateConfig(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("Init Test config failed")
}
// remove the root directory after the test ends.
defer removeAll(rootPath)
configPath := filepath.Join(rootPath, minioConfigFile)
v := v14
testCases := []struct {
configData string
shouldPass bool
}{
// Test 1 - wrong json
{`{`, false},
// Test 2 - empty json
{`{}`, false},
// Test 3 - wrong config version
{`{"version": "10"}`, false},
// Test 4 - wrong browser parameter
{`{"version": "` + v + `", "browser": "foo"}`, false},
// Test 5 - missing credential
{`{"version": "` + v + `", "browser": "on"}`, false},
// Test 6 - missing secret key
{`{"version": "` + v + `", "browser": "on", "credential" : {"accessKey":"minio", "secretKey":""}}`, false},
// Test 7 - missing region should pass, defaults to 'us-east-1'.
{`{"version": "` + v + `", "browser": "on", "credential" : {"accessKey":"minio", "secretKey":"minio123"}}`, true},
// Test 8 - missing browser should pass, defaults to 'on'.
{`{"version": "` + v + `", "region": "us-east-1", "credential" : {"accessKey":"minio", "secretKey":"minio123"}}`, true},
// Test 9 - success
{`{"version": "` + v + `", "browser": "on", "region":"us-east-1", "credential" : {"accessKey":"minio", "secretKey":"minio123"}}`, true},
// Test 10 - duplicated json keys
{`{"version": "` + v + `", "browser": "on", "browser": "on", "region":"us-east-1", "credential" : {"accessKey":"minio", "secretKey":"minio123"}}`, false},
// Test 11 - Wrong Console logger level
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "logger": { "console": { "enable": true, "level": "foo" } }}`, false},
// Test 12 - Wrong File logger level
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "logger": { "file": { "enable": true, "level": "foo" } }}`, false},
// Test 13 - Test AMQP
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "amqp": { "1": { "enable": true, "url": "", "exchange": "", "routingKey": "", "exchangeType": "", "mandatory": false, "immediate": false, "durable": false, "internal": false, "noWait": false, "autoDeleted": false }}}}`, false},
// Test 14 - Test NATS
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "nats": { "1": { "enable": true, "address": "", "subject": "", "username": "", "password": "", "token": "", "secure": false, "pingInterval": 0, "streaming": { "enable": false, "clusterID": "", "clientID": "", "async": false, "maxPubAcksInflight": 0 } } }}}`, false},
// Test 15 - Test ElasticSearch
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "elasticsearch": { "1": { "enable": true, "url": "", "index": "" } }}}`, false},
// Test 16 - Test Redis
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "redis": { "1": { "enable": true, "address": "", "password": "", "key": "" } }}}`, false},
// Test 17 - Test PostgreSQL
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "postgresql": { "1": { "enable": true, "connectionString": "", "table": "", "host": "", "port": "", "user": "", "password": "", "database": "" }}}}`, false},
// Test 18 - Test Kafka
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "kafka": { "1": { "enable": true, "brokers": null, "topic": "" } }}}`, false},
// Test 19 - Test Webhook
{`{"version": "` + v + `", "credential": { "accessKey": "minio", "secretKey": "minio123" }, "region": "us-east-1", "browser": "on", "notify": { "webhook": { "1": { "enable": true, "endpoint": "" } }}}`, false},
}
for i, testCase := range testCases {
if werr := ioutil.WriteFile(configPath, []byte(testCase.configData), 0700); werr != nil {
t.Fatal(werr)
}
verr := validateConfig()
if testCase.shouldPass && verr != nil {
t.Errorf("Test %d, should pass but it failed with err = %v", i+1, verr)
}
if !testCase.shouldPass && verr == nil {
t.Errorf("Test %d, should fail but it succeed.", i+1)
}
}
}
-93
View File
@@ -1,93 +0,0 @@
/*
* Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"os"
"path/filepath"
"sync"
"github.com/minio/go-homedir"
)
// configPath for custom config path only for testing purposes
var customConfigPath string
var configMu sync.Mutex
// Sets a new config path.
func setGlobalConfigPath(configPath string) {
configMu.Lock()
defer configMu.Unlock()
customConfigPath = configPath
}
// getConfigPath get server config path
func getConfigPath() (string, error) {
configMu.Lock()
defer configMu.Unlock()
if customConfigPath != "" {
return customConfigPath, nil
}
homeDir, err := homedir.Dir()
if err != nil {
return "", err
}
configPath := filepath.Join(homeDir, globalMinioConfigDir)
return configPath, nil
}
// mustGetConfigPath must get server config path.
func mustGetConfigPath() string {
configPath, err := getConfigPath()
if err != nil {
return ""
}
return configPath
}
// createConfigPath create server config path.
func createConfigPath() error {
configPath, err := getConfigPath()
if err != nil {
return err
}
return os.MkdirAll(configPath, 0700)
}
// isConfigFileExists - returns true if config file exists.
func isConfigFileExists() bool {
path, err := getConfigFile()
if err != nil {
return false
}
st, err := os.Stat(path)
// If file exists and is regular return true.
if err == nil && st.Mode().IsRegular() {
return true
}
return false
}
// getConfigFile get server config file.
func getConfigFile() (string, error) {
configPath, err := getConfigPath()
if err != nil {
return "", err
}
return filepath.Join(configPath, globalMinioConfigFile), nil
}
+106
View File
@@ -0,0 +1,106 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"fmt"
"net/http"
"net/url"
"strconv"
"strings"
)
// Writes S3 compatible copy part range error.
func writeCopyPartErr(w http.ResponseWriter, err error, url *url.URL) {
switch err {
case errInvalidRange:
writeErrorResponse(w, ErrInvalidCopyPartRange, url)
return
case errInvalidRangeSource:
writeErrorResponse(w, ErrInvalidCopyPartRangeSource, url)
return
default:
writeErrorResponse(w, ErrInternalError, url)
return
}
}
// Parses x-amz-copy-source-range for CopyObjectPart API. Specifically written to
// differentiate the behavior between regular httpRange header v/s x-amz-copy-source-range.
// The range of bytes to copy from the source object. The range value must use the form
// bytes=first-last, where the first and last are the zero-based byte offsets to copy.
// For example, bytes=0-9 indicates that you want to copy the first ten bytes of the source.
// http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
func parseCopyPartRange(rangeString string, resourceSize int64) (hrange *httpRange, err error) {
// Return error if given range string doesn't start with byte range prefix.
if !strings.HasPrefix(rangeString, byteRangePrefix) {
return nil, fmt.Errorf("'%s' does not start with '%s'", rangeString, byteRangePrefix)
}
// Trim byte range prefix.
byteRangeString := strings.TrimPrefix(rangeString, byteRangePrefix)
// Check if range string contains delimiter '-', else return error. eg. "bytes=8"
sepIndex := strings.Index(byteRangeString, "-")
if sepIndex == -1 {
return nil, errInvalidRange
}
offsetBeginString := byteRangeString[:sepIndex]
offsetBegin := int64(-1)
// Convert offsetBeginString only if its not empty.
if len(offsetBeginString) > 0 {
if !validBytePos.MatchString(offsetBeginString) {
return nil, errInvalidRange
}
if offsetBegin, err = strconv.ParseInt(offsetBeginString, 10, 64); err != nil {
return nil, errInvalidRange
}
}
offsetEndString := byteRangeString[sepIndex+1:]
offsetEnd := int64(-1)
// Convert offsetEndString only if its not empty.
if len(offsetEndString) > 0 {
if !validBytePos.MatchString(offsetEndString) {
return nil, errInvalidRange
}
if offsetEnd, err = strconv.ParseInt(offsetEndString, 10, 64); err != nil {
return nil, errInvalidRange
}
}
// rangeString contains first byte positions. eg. "bytes=2-" or
// rangeString contains last bye positions. eg. "bytes=-2"
if offsetBegin == -1 || offsetEnd == -1 {
return nil, errInvalidRange
}
// Last byte position should not be greater than first byte
// position. eg. "bytes=5-2"
if offsetBegin > offsetEnd {
return nil, errInvalidRange
}
// First and last byte positions should not be >= resourceSize.
if offsetBegin >= resourceSize || offsetEnd >= resourceSize {
return nil, errInvalidRangeSource
}
// Success..
return &httpRange{offsetBegin, offsetEnd, resourceSize}, nil
}
+85
View File
@@ -0,0 +1,85 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import "testing"
// Test parseCopyPartRange()
func TestParseCopyPartRange(t *testing.T) {
// Test success cases.
successCases := []struct {
rangeString string
offsetBegin int64
offsetEnd int64
length int64
}{
{"bytes=2-5", 2, 5, 4},
{"bytes=2-9", 2, 9, 8},
{"bytes=2-2", 2, 2, 1},
{"bytes=0000-0006", 0, 6, 7},
}
for _, successCase := range successCases {
hrange, err := parseCopyPartRange(successCase.rangeString, 10)
if err != nil {
t.Fatalf("expected: <nil>, got: %s", err)
}
if hrange.offsetBegin != successCase.offsetBegin {
t.Fatalf("expected: %d, got: %d", successCase.offsetBegin, hrange.offsetBegin)
}
if hrange.offsetEnd != successCase.offsetEnd {
t.Fatalf("expected: %d, got: %d", successCase.offsetEnd, hrange.offsetEnd)
}
if hrange.getLength() != successCase.length {
t.Fatalf("expected: %d, got: %d", successCase.length, hrange.getLength())
}
}
// Test invalid range strings.
invalidRangeStrings := []string{
"bytes=8",
"bytes=5-2",
"bytes=+2-5",
"bytes=2-+5",
"bytes=2--5",
"bytes=-",
"",
"2-5",
"bytes = 2-5",
"bytes=2 - 5",
"bytes=0-0,-1",
"bytes=2-5 ",
}
for _, rangeString := range invalidRangeStrings {
if _, err := parseCopyPartRange(rangeString, 10); err == nil {
t.Fatalf("expected: an error, got: <nil> for range %s", rangeString)
}
}
// Test error range strings.
errorRangeString := []string{
"bytes=10-10",
"bytes=20-30",
}
for _, rangeString := range errorRangeString {
if _, err := parseCopyPartRange(rangeString, 10); err != errInvalidRangeSource {
t.Fatalf("expected: %s, got: %s", errInvalidRangeSource, err)
}
}
}
+86 -21
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -19,22 +19,32 @@ package cmd
import (
"crypto/rand"
"encoding/base64"
"errors"
"github.com/minio/mc/pkg/console"
"golang.org/x/crypto/bcrypt"
)
const (
accessKeyMinLen = 5
accessKeyMaxLen = 20
secretKeyMinLen = 8
secretKeyMaxLen = 40
alphaNumericTable = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
alphaNumericTableLen = byte(len(alphaNumericTable))
accessKeyMinLen = 5
accessKeyMaxLen = 20
secretKeyMinLen = 8
secretKeyMaxLenAmazon = 40
alphaNumericTable = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
alphaNumericTableLen = byte(len(alphaNumericTable))
)
var (
errInvalidAccessKeyLength = errors.New("Invalid access key, access key should be 5 to 20 characters in length")
errInvalidSecretKeyLength = errors.New("Invalid secret key, secret key should be 8 to 40 characters in length")
)
var secretKeyMaxLen = secretKeyMaxLenAmazon
func mustGetAccessKey() string {
keyBytes := make([]byte, accessKeyMaxLen)
if _, err := rand.Read(keyBytes); err != nil {
panic(err)
console.Fatalf("Unable to generate access key. Err: %s.\n", err)
}
for i := 0; i < accessKeyMaxLen; i++ {
@@ -47,7 +57,7 @@ func mustGetAccessKey() string {
func mustGetSecretKey() string {
keyBytes := make([]byte, secretKeyMaxLen)
if _, err := rand.Read(keyBytes); err != nil {
panic(err)
console.Fatalf("Unable to generate secret key. Err: %s.\n", err)
}
return string([]byte(base64.StdEncoding.EncodeToString(keyBytes))[:secretKeyMaxLen])
@@ -65,22 +75,77 @@ func isSecretKeyValid(secretKey string) bool {
// credential container for access and secret keys.
type credential struct {
AccessKey string `json:"accessKey"`
SecretKey string `json:"secretKey"`
AccessKey string `json:"accessKey,omitempty"`
SecretKey string `json:"secretKey,omitempty"`
secretKeyHash []byte
}
func newCredential() credential {
return credential{mustGetAccessKey(), mustGetSecretKey()}
// IsValid - returns whether credential is valid or not.
func (cred credential) IsValid() bool {
return isAccessKeyValid(cred.AccessKey) && isSecretKeyValid(cred.SecretKey)
}
func getCredential(accessKey, secretKey string) (credential, error) {
// Equals - returns whether two credentials are equal or not.
func (cred credential) Equal(ccred credential) bool {
if !ccred.IsValid() {
return false
}
if cred.secretKeyHash == nil {
secretKeyHash, err := bcrypt.GenerateFromPassword([]byte(cred.SecretKey), bcrypt.DefaultCost)
if err != nil {
errorIf(err, "Unable to generate hash of given password")
return false
}
cred.secretKeyHash = secretKeyHash
}
return (cred.AccessKey == ccred.AccessKey &&
bcrypt.CompareHashAndPassword(cred.secretKeyHash, []byte(ccred.SecretKey)) == nil)
}
func createCredential(accessKey, secretKey string) (cred credential, err error) {
if !isAccessKeyValid(accessKey) {
return credential{}, errInvalidAccessKeyLength
err = errInvalidAccessKeyLength
} else if !isSecretKeyValid(secretKey) {
err = errInvalidSecretKeyLength
} else {
var secretKeyHash []byte
secretKeyHash, err = bcrypt.GenerateFromPassword([]byte(secretKey), bcrypt.DefaultCost)
if err == nil {
cred.AccessKey = accessKey
cred.SecretKey = secretKey
cred.secretKeyHash = secretKeyHash
}
}
if !isSecretKeyValid(secretKey) {
return credential{}, errInvalidSecretKeyLength
}
return credential{accessKey, secretKey}, nil
return cred, err
}
// Initialize a new credential object
func mustGetNewCredential() credential {
// Generate access key.
keyBytes := make([]byte, accessKeyMaxLen)
if _, err := rand.Read(keyBytes); err != nil {
console.Fatalln("Unable to generate access key.", err)
}
for i := 0; i < accessKeyMaxLen; i++ {
keyBytes[i] = alphaNumericTable[keyBytes[i]%alphaNumericTableLen]
}
accessKey := string(keyBytes)
// Generate secret key.
keyBytes = make([]byte, secretKeyMaxLen)
if _, err := rand.Read(keyBytes); err != nil {
console.Fatalln("Unable to generate secret key.", err)
}
secretKey := string([]byte(base64.StdEncoding.EncodeToString(keyBytes))[:secretKeyMaxLen])
cred, err := createCredential(accessKey, secretKey)
if err != nil {
console.Fatalln("Unable to generate new credential.", err)
}
return cred
}
+101
View File
@@ -0,0 +1,101 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import "testing"
func TestMustGetNewCredential(t *testing.T) {
cred := mustGetNewCredential()
if !cred.IsValid() {
t.Fatalf("Failed to get new valid credential")
}
}
func TestCreateCredential(t *testing.T) {
cred := mustGetNewCredential()
testCases := []struct {
accessKey string
secretKey string
expectedResult bool
expectedErr error
}{
// Access key too small.
{"user", "pass", false, errInvalidAccessKeyLength},
// Access key too long.
{"user12345678901234567", "pass", false, errInvalidAccessKeyLength},
// Access key contains unsuppported characters.
{"!@#$", "pass", false, errInvalidAccessKeyLength},
// Secret key too small.
{"myuser", "pass", false, errInvalidSecretKeyLength},
// Secret key too long.
{"myuser", "pass1234567890123456789012345678901234567", false, errInvalidSecretKeyLength},
// Success when access key contains leading/trailing spaces.
{" user ", cred.SecretKey, true, nil},
{"myuser", "mypassword", true, nil},
{cred.AccessKey, cred.SecretKey, true, nil},
}
for _, testCase := range testCases {
cred, err := createCredential(testCase.accessKey, testCase.secretKey)
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
} else if err == nil {
t.Fatalf("error: expected = %v, got = <nil>", testCase.expectedErr)
} else if testCase.expectedErr.Error() != err.Error() {
t.Fatalf("error: expected = %v, got = %v", testCase.expectedErr, err)
}
if testCase.expectedResult != cred.IsValid() {
t.Fatalf("cred: expected: %v, got: %v", testCase.expectedResult, cred.IsValid())
}
}
}
func TestCredentialEqual(t *testing.T) {
cred := mustGetNewCredential()
testCases := []struct {
cred credential
ccred credential
expectedResult bool
}{
// Empty compare credential
{cred, credential{}, false},
// Empty credential
{credential{}, cred, false},
// Two different credentials
{cred, mustGetNewCredential(), false},
// Access key is different in compare credential.
{cred, credential{AccessKey: "myuser", SecretKey: cred.SecretKey}, false},
// Secret key is different in compare credential.
{cred, credential{AccessKey: cred.AccessKey, SecretKey: "mypassword"}, false},
// secretHashKey is missing in compare credential.
{cred, credential{AccessKey: cred.AccessKey, SecretKey: cred.SecretKey}, true},
// secretHashKey is missing in credential.
{credential{AccessKey: cred.AccessKey, SecretKey: cred.SecretKey}, cred, true},
// Same credentials.
{cred, cred, true},
}
for _, testCase := range testCases {
result := testCase.cred.Equal(testCase.ccred)
if result != testCase.expectedResult {
t.Fatalf("cred: expected: %v, got: %v", testCase.expectedResult, result)
}
}
}
+5 -6
View File
@@ -28,7 +28,7 @@ import (
// erasureCreateFile - writes an entire stream by erasure coding to
// all the disks, writes also calculate individual block's checksum
// for future bit-rot protection.
func erasureCreateFile(disks []StorageAPI, volume, path string, reader io.Reader, blockSize int64, dataBlocks int, parityBlocks int, algo string, writeQuorum int) (bytesWritten int64, checkSums []string, err error) {
func erasureCreateFile(disks []StorageAPI, volume, path string, reader io.Reader, allowEmpty bool, blockSize int64, dataBlocks int, parityBlocks int, algo string, writeQuorum int) (bytesWritten int64, checkSums []string, err error) {
// Allocated blockSized buffer for reading from incoming stream.
buf := make([]byte, blockSize)
@@ -47,7 +47,7 @@ func erasureCreateFile(disks []StorageAPI, volume, path string, reader io.Reader
// We have reached EOF on the first byte read, io.Reader
// must be 0bytes, we don't need to erasure code
// data. Will create a 0byte file instead.
if bytesWritten == 0 {
if bytesWritten == 0 && allowEmpty {
blocks = make([][]byte, len(disks))
rErr = appendFile(disks, volume, path, blocks, hashWriters, writeQuorum)
if rErr != nil {
@@ -114,6 +114,7 @@ func appendFile(disks []StorageAPI, volume, path string, enBlocks [][]byte, hash
// Write encoded data to quorum disks in parallel.
for index, disk := range disks {
if disk == nil {
wErrs[index] = traceError(errDiskNotFound)
continue
}
wg.Add(1)
@@ -123,6 +124,8 @@ func appendFile(disks []StorageAPI, volume, path string, enBlocks [][]byte, hash
wErr := disk.AppendFile(volume, path, enBlocks[index])
if wErr != nil {
wErrs[index] = traceError(wErr)
// Ignore disk which returned an error.
disks[index] = nil
return
}
@@ -137,9 +140,5 @@ func appendFile(disks []StorageAPI, volume, path string, enBlocks [][]byte, hash
// Wait for all the appends to finish.
wg.Wait()
// Do we have write quorum?.
if !isDiskQuorum(wErrs, writeQuorum) {
return traceError(errXLWriteQuorum)
}
return reduceWriteQuorumErrs(wErrs, objectOpIgnoredErrs, writeQuorum)
}
+4 -4
View File
@@ -56,7 +56,7 @@ func TestErasureCreateFile(t *testing.T) {
t.Fatal(err)
}
// Test when all disks are up.
size, _, err := erasureCreateFile(disks, "testbucket", "testobject1", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
size, _, err := erasureCreateFile(disks, "testbucket", "testobject1", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
@@ -69,7 +69,7 @@ func TestErasureCreateFile(t *testing.T) {
disks[5] = AppendDiskDown{disks[5].(*posix)}
// Test when two disks are down.
size, _, err = erasureCreateFile(disks, "testbucket", "testobject2", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
size, _, err = erasureCreateFile(disks, "testbucket", "testobject2", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
@@ -83,7 +83,7 @@ func TestErasureCreateFile(t *testing.T) {
disks[8] = AppendDiskDown{disks[8].(*posix)}
disks[9] = AppendDiskDown{disks[9].(*posix)}
size, _, err = erasureCreateFile(disks, "testbucket", "testobject3", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
size, _, err = erasureCreateFile(disks, "testbucket", "testobject3", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
@@ -93,7 +93,7 @@ func TestErasureCreateFile(t *testing.T) {
// 1 more disk down. 7 disk down in total. Should return quorum error.
disks[10] = AppendDiskDown{disks[10].(*posix)}
_, _, err = erasureCreateFile(disks, "testbucket", "testobject4", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
_, _, err = erasureCreateFile(disks, "testbucket", "testobject4", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if errorCause(err) != errXLWriteQuorum {
t.Errorf("erasureCreateFile return value: expected errXLWriteQuorum, got %s", err)
}
+1 -1
View File
@@ -48,7 +48,7 @@ func TestErasureHealFile(t *testing.T) {
t.Fatal(err)
}
// Create a test file.
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject1", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject1", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
+2 -2
View File
@@ -192,7 +192,7 @@ func erasureReadFile(writer io.Writer, disks []StorageAPI, volume string, path s
}()
// Total bytes written to writer
bytesWritten := int64(0)
var bytesWritten int64
startBlock := offset / blockSize
endBlock := (offset + length) / blockSize
@@ -263,7 +263,7 @@ func erasureReadFile(writer io.Writer, disks []StorageAPI, volume string, path s
}
// Offset in enBlocks from where data should be read from.
enBlocksOffset := int64(0)
var enBlocksOffset int64
// Total data to be read from enBlocks.
enBlocksLength := curBlockSize
+4 -34
View File
@@ -121,35 +121,6 @@ func testGetReadDisks(t *testing.T, xl *xlObjects) {
}
}
// Test getOrderedDisks which returns ordered slice of disks from their
// actual distribution.
func testGetOrderedDisks(t *testing.T, xl *xlObjects) {
disks := xl.storageDisks
distribution := []int{16, 14, 12, 10, 8, 6, 4, 2, 1, 3, 5, 7, 9, 11, 13, 15}
orderedDisks := getOrderedDisks(distribution, disks)
// From the "distribution" above you can notice that:
// 1st data block is in the 9th disk (i.e distribution index 8)
// 2nd data block is in the 8th disk (i.e distribution index 7) and so on.
if orderedDisks[0] != disks[8] ||
orderedDisks[1] != disks[7] ||
orderedDisks[2] != disks[9] ||
orderedDisks[3] != disks[6] ||
orderedDisks[4] != disks[10] ||
orderedDisks[5] != disks[5] ||
orderedDisks[6] != disks[11] ||
orderedDisks[7] != disks[4] ||
orderedDisks[8] != disks[12] ||
orderedDisks[9] != disks[3] ||
orderedDisks[10] != disks[13] ||
orderedDisks[11] != disks[2] ||
orderedDisks[12] != disks[14] ||
orderedDisks[13] != disks[1] ||
orderedDisks[14] != disks[15] ||
orderedDisks[15] != disks[0] {
t.Errorf("getOrderedDisks returned incorrect order.")
}
}
// Test for isSuccessDataBlocks and isSuccessDecodeBlocks.
func TestIsSuccessBlocks(t *testing.T) {
dataBlocks := 8
@@ -217,7 +188,7 @@ func TestIsSuccessBlocks(t *testing.T) {
}
}
// Wrapper function for testGetReadDisks, testGetOrderedDisks.
// Wrapper function for testGetReadDisks, testShuffleDisks.
func TestErasureReadUtils(t *testing.T) {
nDisks := 16
disks, err := getRandomDisks(nDisks)
@@ -236,7 +207,6 @@ func TestErasureReadUtils(t *testing.T) {
defer removeRoots(disks)
xl := objLayer.(*xlObjects)
testGetReadDisks(t, xl)
testGetOrderedDisks(t, xl)
}
// Simulates a faulty disk for ReadFile()
@@ -271,7 +241,7 @@ func TestErasureReadFileDiskFail(t *testing.T) {
}
// Create a test file to read from.
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
@@ -354,7 +324,7 @@ func TestErasureReadFileOffsetLength(t *testing.T) {
}
// Create a test file to read from.
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
@@ -433,7 +403,7 @@ func TestErasureReadFileRandomOffsetLength(t *testing.T) {
iterations := 10000
// Create a test file to read from.
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject", bytes.NewReader(data), blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
size, checkSums, err := erasureCreateFile(disks, "testbucket", "testobject", bytes.NewReader(data), true, blockSize, dataBlocks, parityBlocks, bitRotAlgo, dataBlocks+1)
if err != nil {
t.Fatal(err)
}
+5 -5
View File
@@ -24,8 +24,8 @@ import (
"sync"
"github.com/klauspost/reedsolomon"
"github.com/minio/blake2b-simd"
"github.com/minio/sha256-simd"
"golang.org/x/crypto/blake2b"
)
// newHashWriters - inititialize a slice of hashes for the disk count.
@@ -48,14 +48,14 @@ func newHash(algo string) (h hash.Hash) {
// ignore the error, because New512 without a key never fails
// New512 only returns a non-nil error, if the length of the passed
// key > 64 bytes - but we use blake2b as hash function (no key)
h = blake2b.New512()
h, _ = blake2b.New512(nil)
// Add new hashes here.
default:
// Default to blake2b.
// ignore the error, because New512 without a key never fails
// New512 only returns a non-nil error, if the length of the passed
// key > 64 bytes - but we use blake2b as hash function (no key)
h = blake2b.New512()
h, _ = blake2b.New512(nil)
}
return h
}
@@ -116,7 +116,7 @@ func writeDataBlocks(dst io.Writer, enBlocks [][]byte, dataBlocks int, offset in
write := length
// Counter to increment total written.
totalWritten := int64(0)
var totalWritten int64
// Write all data blocks to dst.
for _, block := range enBlocks[:dataBlocks] {
@@ -180,7 +180,7 @@ func copyBuffer(writer io.Writer, disk StorageAPI, volume string, path string, b
}
// Starting offset for Reading the file.
startOffset := int64(0)
var startOffset int64
// Read until io.EOF.
for {
+2 -2
View File
@@ -86,10 +86,10 @@ func traceError(e error, errs ...error) error {
fn := runtime.FuncForPC(pc)
file, line := fn.FileLine(pc)
name := fn.Name()
if strings.HasSuffix(name, "ServeHTTP") {
if hasSuffix(name, "ServeHTTP") {
break
}
if strings.HasSuffix(name, "runtime.") {
if hasSuffix(name, "runtime.") {
break
}

Some files were not shown because too many files have changed in this diff Show More