Compare commits

...

430 Commits

Author SHA1 Message Date
Brendan Ashworth aeafe668d8 posix: do not upstream errors in deleteFile (#4771)
This commit changes posix's deleteFile() to not upstream errors from
removing parent directories. This fixes a race condition.

The race condition occurs when multiple deleteFile()s are called on the
same parent directory, but different child files. Because deleteFile()
recursively removes parent directories if they are empty, but
deleteFile() errors if the selected deletePath does not exist, there was
an opportunity for a race condition. The two processes would remove the
child directories successfully, then depend on the parent directory
still existing. In some cases this is an invalid assumption, because
other processes can remove the parent directory beforehand. This commit
changes deleteFile() to not upstream an error if one occurs, because the
only required error should be from the immediate deletePath, not from a
parent path.

In the specific bug report, multiple CompleteMultipartUpload requests
would launch multiple deleteFile() requests. Because they chain up on
parent directories, ultimately at the end, there would be multiple
remove files for the ultimate parent directory,
.minio.sys/multipart/{bucket}. Because only one will succeed and one
will fail, an error would be upstreamed saying that the file does not
exist, and the CompleteMultipartUpload code interpreted this as
NoSuchKey, or that the object/part id doesn't exist. This was faulty
behavior and is now fixed.

The added test fails before this change and passes after this change.

Fixes: https://github.com/minio/minio/issues/4727
2017-08-04 16:51:20 -07:00
Krishnan Parthasarathi 54f3a0946f Avoid superfluous error messages after connect (#4762)
Peek could fail legitimately when clients abruptly close connection. So,
io.EOF and network timeout errors are not logged while all other errors
will be logged.
2017-08-04 14:35:07 -07:00
Krishna Srinivas be6bd52978 fs: use keymarker and uploadidmarker in the multipart purging loop (#4775)
related to #4564
2017-08-04 14:14:23 -07:00
Krishnan Parthasarathi 75c43bfb6c ListMultipartUploads, ListObjectParts return empty response (#4694)
Also, periodically removes incomplete multipart uploads older than 2 weeks.
2017-08-04 10:45:57 -07:00
ebozduman 0aca2ab970 Stop attempting to close nil Listener (#4753) 2017-08-04 10:44:46 -07:00
Brendan Ashworth 28bc5899fd posix: test isDirEmpty, change error conditional (#4743)
This commit adds a new test for isDirEmpty (for code coverage) and
changes around the error conditional. Previously, there was a `return
nil` statement that would only be triggered under a race condition and
would trip up our test coverage for no real reason. With this new error
conditional, there's no awkward 'else'-esque condition, which means test
coverage will not change between runs for no reason in this specific
test. It's also a cleaner read.
2017-08-04 10:43:51 -07:00
Nitish Tiwari fcc61fa46a Remove minimum inodes reqd check (#4747) 2017-08-03 20:07:22 -07:00
Nitish Tiwari f7889e1af7 Add TLS request for healthcheck (#4740) 2017-08-03 20:05:45 -07:00
Brendan Ashworth bccc386994 fs: drop Stat() call from fsDeleteFile,deleteFile (#4744)
This commit makes fsDeleteFile() simply call deleteFile() after calling
the relevant path length checking functions. This DRYs the code base.

This commit removes the Stat() call from deleteFile(). This improves
performance and removes any possibility of a race condition.

This additionally adds tests and a benchmark for said function. The
results aren't very consistent, although I'd expect this commit to make
it faster.
2017-08-03 20:04:28 -07:00
ebozduman 0f401b67ad Removes max limit requirement on accessKey and secretKey length (#4730) 2017-08-03 20:03:37 -07:00
Laurentiu Nicola 108decfa76 Fix sysctl proposed values (#4741)
`sched_wakeup_granularity_ns` and `sched_wakeup_granularity_ns` are measured in `ns`, so a value of `10` or `15` is way too low.
2017-08-02 14:08:58 -07:00
Nitish Tiwari 6afbd502e8 Add Docker command in erasure code document (#4735) 2017-07-31 11:40:14 -07:00
Brendan Ashworth ec5293ce29 jwt,browser: allow short-expiry tokens for GETs (#4684)
This commit fixes a potential security issue, whereby a full-access
token to the server would be available in the GET URL of a download
request. This fixes that issue by introducing short-expiry tokens, which
are only valid for one minute, and are regenerated for every download
request.

This commit specifically introduces the short-lived tokens, adds tests
for the tokens, adds an RPC call for generating a token given a
full-access token, updates the browser to use the new tokens for
requests where the token is passed as a GET parameter, and adds some
tests with the new temporary tokens.

Refs: https://github.com/minio/minio/pull/4673
2017-07-24 12:46:37 -07:00
Harshavardhana 4785555d34 api: Upon bucket delete remove in-memory state properly. (#4716)
This PR fixes the issue of cleaning up in-memory state
properly. Without this PR we can lead to security
situations where new bucket would inherit wrong
permissions on bucket and expose objects erroneously.

Fixes #4714
2017-07-23 19:35:18 -07:00
A. Elleuch b918a6592f gcs: Better parsing of address flag (#4709) 2017-07-20 16:39:11 -07:00
Krishna Srinivas eb787d8613 gateway-gcs: remove files older than 2 weeks in minio.sys.temp (#4599).
Rename ##minio## to {minio}.
2017-07-20 15:36:48 -07:00
Aaron Kunz 0a1501bc1b Fix typo (#4695) 2017-07-19 15:19:03 -07:00
Harshavardhana f8bd9cfd83 rpc: Do not use read/write deadlines for rpc connections. (#4647)
Fixes #4626
2017-07-18 09:30:46 -07:00
Brendan Ashworth c59b995f7b build: ditch verifiers on make (#4679)
This commit ditches running verifiers automatically when just building
the server. It retains the verifiers when running tests.

There is very little point to running the verifiers each time a
developer builds the library but has no intent of running the tests.
They're expensive in time; this commit halves the build time on my
system, from 57 seconds to 29 seconds. This is because verifiers updates
the libraries from GitHub each time, which is slightly wasteful.
Additionally, computing cyclomatic complexity is expensive
computationally and isn't necessary to build the library.

Additionally, this allows the library to be built offline. It no longer
requires internet to run make.
2017-07-15 12:12:03 -07:00
Harshavardhana bc73a1a1cb gcs: Save partNumber as part of backend format. (#4666)
Fixes #4637
2017-07-13 23:20:16 -07:00
Krishna Srinivas ce7c9c651d gateway-azure: Return right error when Part size is > 100MB (#4652) 2017-07-12 16:42:14 -07:00
Bala FA c3dd7c1f6c Refactor HTTP server to address bugs (#4636)
* Refactor HTTP server to address bugs
* Remove unnecessary goroutine to start multiple TCP listeners.
* HTTP server waits for shutdown to maximum of Server.ShutdownTimeout
  than per serverShutdownPoll.
* Handles new connection errors properly.
* Handles read and write timeout properly.
* Handles error on start of HTTP server properly by exiting minio
  process.

Fixes #4494 #4476 & fixed review comments
2017-07-12 16:33:21 -07:00
Harshavardhana 2d23cd4f39 gcs: Fetch port as GlobalString(). (#4657)
Currently we were looking for `address` flag
under local flags. This PR fixes #4656
2017-07-11 18:06:26 -07:00
Harshavardhana ce7af3aae1 gcs: Fix writer/reader go-routine leaks and code re-use (#4651)
This PR serves to fix following things in GCS gateway.

- fixes leaks in object reader and writer, not getting closed
  under certain situations. This led to go-routine leaks.

- apparent confusing issue in case of complete multipart upload,
  where it is currently possible for an entirely different
  object name to concatenate parts of a different object name
  if you happen to know the upload-id and parts of the object.
  This is a very rare scenario but it is possible.

- succint usage of certain parts of code base and re-use.
2017-07-11 09:25:19 -07:00
Krishna Srinivas 1b92c5136b Append "-1" to etag when it is not MD5 (#4641)
* gateway-azure: append "-1" to ETag so that clients do not interpret it as MD5. fixes #4537. Added unit tests.
2017-07-10 18:21:12 -07:00
Harshavardhana cc8a8cb877 posix: Check for min disk space and inodes (#4618)
This is needed such that we don't start or
allow writing to a posix disk which doesn't
have minimum total disk space available.

One part fix for #4617
2017-07-10 18:14:48 -07:00
Krishna Srinivas ce403fdaa0 GCS documentation (#4622)
* GCS documentation and review fixes.
2017-07-10 09:35:20 -07:00
Nitish Tiwari 45fbb0d618 Add NATS Streaming doc to event notification doc (#4645) 2017-07-07 23:37:12 -07:00
Harshavardhana f5ce685aa1 Remove dead unused errs and constants. (#4627) 2017-07-07 14:31:42 -07:00
Krishna Srinivas c83055500d fs: Fail CompleteMultipartUpload if partSize < 5M unless it is last part (#4642)
fixes #4625
2017-07-07 08:41:29 -07:00
Andreas Auernhammer b0fbddc051 fix confusing code for http.Header handling (#4623)
Fixed header-to-metadat extraction. The extractMetadataFromHeader function should return an error if the http.Header contains a non-canonicalized key. The reason is that the keys can be manually set (through a map access) which can lead to ugly bugs.
Also fixed header-to-metadata extraction. Return a InternalError if a non-canonicalized key is found in a http.Header. Also log the error.
2017-07-05 16:56:10 -07:00
Harshavardhana 4e0c08e9c5 ListenBucketNotification should set proper MIME type. (#4621)
This is needed to avoid proxies buffering the connection
this is also a HTTP standard way to handle this situation
where server is sending back events in asynchronously.

For more details read https://goo.gl/RCML9f

Fixes - https://github.com/minio/minio-go/issues/731
2017-07-03 19:59:41 -07:00
Nitish Tiwari 344f9ec608 Fix gateway browser screenshot (#4613) 2017-06-30 09:47:40 -07:00
Dee Koder 28ff62716f Fixed one of the images with missing details. (#4612) 2017-06-29 15:49:09 -07:00
Dee Koder 1f69a75efa Updated docs with latest images. (#4611) 2017-06-29 11:41:21 -07:00
Nitish Tiwari 02a81ee564 Remove deployment scenarios from erasure code guide (#4607) 2017-06-29 11:40:59 -07:00
Nitish Tiwari e91e9e8a38 GCS ListObjectV2 honours continuationToken (#4608) 2017-06-29 11:19:55 -07:00
Dee Koder e45f9057d6 Added new optimized images. (#4609) 2017-06-29 09:10:47 -07:00
Bala FA 53e7fdc847 create subcomposed objects if total parts are > 32 (#4593) 2017-06-27 22:27:05 -07:00
Aditya Manthramurthy 1af331c05c Remove unnecessary newline at beginning of server output (#4600) 2017-06-27 19:46:58 -07:00
ebozduman 0e5b9c7fe4 Adds 'gcs not ready for production' msg (#4604) 2017-06-27 19:44:47 -07:00
Harshavardhana a86dc8a4c5 cleanup makefile and enable CGO_ENABLED=0 (#4598) 2017-06-26 18:07:06 -07:00
A. Elleuch c88dca984d web: Encode path in presigned GET urls (#4596)
When the browser asks for a GET presigned url, this latter is not
encoded and can be confusing when the user copies-pastes it somewhere,
especially when the path contains a space.
2017-06-25 18:39:14 -07:00
Harshavardhana 1054f9cbf0 fix: Remove adverb from erasure coded startup message. (#4594) 2017-06-25 18:38:55 -07:00
Harshavardhana aaacce4c43 browser: update ui-assets with new changes. (#4595) 2017-06-25 18:09:24 -07:00
Nitish Tiwari 7bd1f44491 Add support for helm package info in useragent (#4592) 2017-06-24 13:17:28 -07:00
Harshavardhana ba6d997d18 build: Error out properly when unsupported arch is given. (#4585) 2017-06-24 01:05:35 -07:00
A. Elleuch eaa41e4086 gcs: Check if the given project id argument exists (#4583)
Using GCS resource manager API, check if the provided
project id is already created and associated to the current
user account.
2017-06-23 22:10:29 -07:00
Krishna Srinivas 6b70f429ed gateway/azure: Parse error responses for anonymous requests (#4543)
fixes #4481
2017-06-23 22:07:46 -07:00
Krishna Srinivas 0a6e9a1834 gateway-gcs: cleanup minio.sys.temp before deleting the bucket (#4582)
fixes #4560
fixes #4569
2017-06-23 17:57:25 -07:00
Nitish Tiwari 15b65a8342 Added AnonListObjectsV2 support to GCS (#4584) 2017-06-23 17:35:45 -07:00
Harshavardhana 8b7df7da37 api: No need to set x-amz-bucket-region if region is empty. (#4586) 2017-06-23 16:05:40 -07:00
Krishnan Parthasarathi 237c8af5ef Improve GCS gateway example instruction (#4587) 2017-06-23 13:13:26 -07:00
Krishna Srinivas ff036c171f gateway-gcs: double quotes should be striped from ETag (#4590) 2017-06-23 12:19:10 -07:00
Harshavardhana a3b085300d gcs: Add missing AnonListObjectsV2 2017-06-22 12:09:13 -07:00
Harshavardhana b90cefdb88 Merge remote-tracking branch 'origin/feature-gcs' 2017-06-22 11:52:12 -07:00
Harshavardhana f3506b8958 tests: Enable previously disabled UNC tests on CI. (#4575)
Windows CI had a bug which has been fixed, re-enable
those commented tests.
2017-06-22 07:54:22 -07:00
Frank Wessels 46897b1100 Name return values to prevent the need (and unnecessary code bloat) (#4576)
This is done to explicitly instantiate objects for every return statement.
2017-06-21 19:53:09 -07:00
Harshavardhana cec8b238f3 sign: StreamingSign should use region from client. (#4577)
This is a fix to make streaming signature to behave
the same as regular signature and presigned signature.

Fixes https://github.com/minio/minio-go/issues/718
2017-06-21 11:30:34 -07:00
Krishna Srinivas 13ab8e17e2 gateway-gcs: use minio.sys.temp/multipart/v1 as url base (#4562) 2017-06-21 10:27:44 -07:00
Nitish Tiwari 9d3d64df2d Added HEALTHCHECK in release Dockerfiles (#4550)
A new script `healthcheck.sh` checks for http status 200/404
2017-06-20 19:24:45 -07:00
Krishnan Parthasarathi 146bc3e638 Add MINIO_REGION to server help message (#4558)
* Add e.g for setting MINIO_REGION env variable
* Add MINIO_REGION to region table
2017-06-20 15:02:18 -07:00
Harshavardhana 0543d45fb3 fix: OwnerID in response should be 64 character in length. (#4554)
Rather than sending a custom "minio" string, we can
change this to `sha256('arn:aws:iam::minio:user/admin')`.

Fixes #4553
2017-06-20 15:01:13 -07:00
Krishnan Parthasarathi fe426944ea Fix GCS help message (#4570) 2017-06-20 14:25:16 -07:00
Aditya Manthramurthy c1a6ca0c33 Fix spelling of function name to startLockMaintenance (#4561) 2017-06-20 12:10:02 -07:00
Harshavardhana 5a78266821 gateway/gcs: Complete minio browser support for gcs. (#4552)
Fixes #4460
2017-06-19 19:45:13 -07:00
Harshavardhana f5b4b0765a Update minio-go dependency (#4551)
This updates dependency for

 - AWS S3 backend.
 - pkg/madmin

```
- Relax isValidBucketName to allow reading existing buckets. (#708) (3 minutes ago) <Harshavardhana>
- For GCS the size limit of S3 is not useful. (#711) (3 days ago) <Harshavardhana>
- s3utils: Support AWS S3 US GovCloud endpoint. (#701) (3 days ago) <Harshavardhana>
- api: Always strip 80/443 port from host (#709) (3 days ago) <Anis Elleuch>
- Redact signature strings properly. (#706) (4 days ago) <Harshavardhana>
- api: Single putObject can use temporary file always. (#703) (6 days ago) <Harshavardhana>
- Spelling fix (#704) (7 days ago) <Jacob Taylor>
- api/encrypt: Get() on encrypted object should be a reader. (#699) (2 weeks ago) <Harshavardhana>
- get: Fix reading an object if its size is unknown (#694) (3 weeks ago) <Anis Elleuch>
- fixes #696 by updating the examples for put-encrypted-object and get-encrypted-object (#697) (3 weeks ago) <Tejay Cardon>
- fix InvalidAccessKeyId error according to amazon documentation (#692) (4 weeks ago) <samkevich>
- Add AWS S3 SSE-C example. (#689) (4 weeks ago) <Harshavardhana>
- According to RFC7232 Etag should be in quotes for If-Match. (#688) (5 weeks ago) <Harshavardhana>
- api: getReaderSize() should honor seeked file descriptors. (#681) (5 weeks ago) <Harshavardhana>
- tests: Use bytes.Repeat() when generating big data (#683) (5 weeks ago) <Anis Elleuch>
- api: Failed call retry with region only when http.StatusBadRequest. (#678) (5 weeks ago) <Harshavardhana>
- api: Add NewWithCredentials() (#646) (5 weeks ago) <Harshavardhana>
```
2017-06-19 16:02:35 -07:00
Krishna Srinivas 3928c1e14c gateway/gcs: Change in multipart backend format (#4455) 2017-06-17 16:00:41 -07:00
Krishnan Parthasarathi 94241cd153 Update Minio on DC/OS doc for latest Minio package (#4549) 2017-06-17 12:21:37 -07:00
Harshavardhana a86c2e2ce1 xl/fs: Return InvalidPart{} error for part ETag mismatch. (#4541)
Fixes #4539
2017-06-17 11:20:39 -07:00
Harshavardhana e99244be02 xl: prepare storage should Abort properly. (#4542)
Current state-machine didn't honor a situation
which can arise when there is a combination of

 - formatted
 - unformatted
 - corrupted

disks - this combination invariably goes into a
mode where all servers are waiting perpetually
forever thinking we will get quorum in future.

At this point there is a distant possibility of
ever getting a quorum since we don't even have
quorum number of disks offline.

We should exit and print a proper message per disk
to indicate what went wrong and what was detected
by the server.

Refer #4477
2017-06-17 11:20:12 -07:00
Nitish Tiwari 58833711e0 Added ListObjectsV2 and ListObjectsV2 Anon support to Gateway S3 and Azure. (#4547) 2017-06-16 22:17:00 -07:00
Harshavardhana f99f218999 Add support for reading and saving config on Gateway. (#4463)
This is also a first step towards supporting bucket
notification for gateway.
2017-06-16 16:01:41 -07:00
Krishnan Parthasarathi 4fb5fc72d7 GCS gateway allows apps to supply their own marker (#4495)
Most s3 compatible apps use object keys returned in listing as
marker. This change allows this behaviour with gateway-gcs too.
2017-06-16 15:02:07 -07:00
Remco Verhoef d86973dcca Allow bucket creation in different regions, closes #4287 and #4241
* I needed to remove the region check from PutBucketHandler
2017-06-16 15:02:07 -07:00
Krishnan Parthasarathi 8085ba4494 Filter out internal object prefix during listing (#4435)
We use ZZZZ-Minio/ prefix internally in our GCS gateway which should be
filtered out in the response to ListObjects.
2017-06-16 15:02:07 -07:00
poornas 9bd0eb1a9e Set default ETag value if vendor returns empty md5 string (#4409)
The ETag is constructed from md5 atttribute of object attributes
returned by the vendor's Composer. The md5 attribute comes back
as nil for large uploads. Instead the CRC32C should be used.

Refer to https://cloud.google.com/storage/docs/hashes-etags

Fixes #4397
2017-06-16 15:02:07 -07:00
Anis Elleuch e4e0abfc05 fix: Check project id before starting gateway (#4412) 2017-06-16 15:02:07 -07:00
poornas 12b2fc894b Remove profile option for gcs from gateway help message (#4421) 2017-06-16 15:01:34 -07:00
Krishna Srinivas 2aa76e7407 Change md5Sum to etag (#4399) 2017-06-16 14:58:49 -07:00
Remco Verhoef 0dab038858 Cleanup and update the PR with the master branch. 2017-06-16 14:55:32 -07:00
Remco Verhoef a76556ec1b Map only default region us-east-1 to gcs us region 2017-06-16 14:54:37 -07:00
Harshavardhana 91c7bb65c5 gateway/gcs: send proper error responses for Get/SetBucket policies. (#4338)
Fixes #4323
2017-06-16 14:54:37 -07:00
Anis Elleuch 5d602034ea gateway: Use default params when no args provided (#4315)
For S3 & Azure, use default parameters when no arguments (endpoint) are
provided. This also avoids a crash.
2017-06-16 14:54:37 -07:00
Nitish Tiwari b829ec4a6b Fixes https://github.com/minio/minio/issues/4320 (#4332)
- Add description for error ErrBucketAlreadyExists
2017-06-16 14:54:37 -07:00
Remco Verhoef 9c50a9f567 Fix ListObjectParts to list properly all parts - closes #4322 2017-06-16 14:54:37 -07:00
Remco Verhoef 52122c0309 Fix uploadIDMarker handling. 2017-06-16 14:54:37 -07:00
Remco Verhoef 3b9d313c87 Fix issue with AbortMultipartUpload, closes #4322 2017-06-16 14:54:37 -07:00
Remco Verhoef bfff251e2a Fix issue with UNSIGNED payloads.
Additionally also fixes escaping slashes in
temporary multipart names
2017-06-16 14:54:37 -07:00
Remco Verhoef 52b500cce9 Verify multipart etag during complete, closes #4288 2017-06-16 14:54:37 -07:00
Remco Verhoef c63cdca11f Support iterating through ListObjectParts using NextPartNumberMarker, closes #4284 2017-06-16 14:54:37 -07:00
Remco Verhoef 4430085981 Add region to gcs gateway example 2017-06-16 14:54:37 -07:00
Remco Verhoef 5c78415b31 Verify md5 content hash, closes #4285 2017-06-16 14:54:37 -07:00
Remco Verhoef bd67117756 Use maxKeys for iterator 2017-06-16 14:54:37 -07:00
Remco Verhoef f3e5e9fb29 Support marker, closes #4286 2017-06-16 14:54:37 -07:00
Remco Verhoef 2de1921fe8 Use MINIO_REGION environment variable for region configuration, closes #4287 2017-06-16 14:54:37 -07:00
Remco Verhoef dd7e47f264 Add access and secret key to example, needed to access Minio Gateway 2017-06-16 14:54:37 -07:00
Remco Verhoef fe9d826bef Implement bucket policies 2017-06-16 14:53:36 -07:00
Remco Verhoef 6dbc5aba09 Return correct error when PutObject fails 2017-06-16 14:53:36 -07:00
Remco Verhoef de5374f74c Map S3 regions to Google (multi)regions 2017-06-16 14:53:36 -07:00
Remco Verhoef bf55591c64 Make every backend responsible for parsing its own arguments, fixes #4293 2017-06-16 14:53:36 -07:00
Remco Verhoef 2d814e340f Return BucketAlreadyExists when bucket exists with another user 2017-06-16 14:53:36 -07:00
Remco Verhoef 0a8cf1a6b0 Allow bucket creation in different regions, closes #4287 and #4241
* I needed to remove the region check from PutBucketHandler
2017-06-16 14:53:36 -07:00
Remco Verhoef 07949f68d8 Translate gcs errors to S3 compatible errors, fixes #4278 2017-06-16 14:53:36 -07:00
Remco Verhoef 909a89647b Use default endpoint when not supplied 2017-06-16 14:53:36 -07:00
Remco Verhoef 6508da5fde Add usage for GCS gateway, closes #4280 2017-06-16 14:53:36 -07:00
Remco Verhoef 3379f005a5 Initial implementation of Google Cloud Storage 2017-06-16 14:47:02 -07:00
Remco Verhoef 4be609eb82 Added AllAccessDisabled error 2017-06-16 14:47:02 -07:00
Remco Verhoef dd5b975001 Add comment, gateway should validate object name 2017-06-16 14:47:02 -07:00
Remco Verhoef 9ac3538141 Move anonymous error to object translation from Azure specific to gateway 2017-06-16 14:47:02 -07:00
Remco ace4f9fd15 Implement gateway support Google Cloud Storage 2017-06-16 14:47:02 -07:00
Nitish Tiwari b283a2c21f Bump docs references to latest Minio release RELEASE.2017-06-13T19-01-01Z (#4546) 2017-06-15 14:20:36 -07:00
Anand Babu (AB) Periasamy ed37bf3703 Reduce macOS instructions 2017-06-14 23:26:33 -07:00
Rushan d0f18dc1b5 browser: Use custom input number step-up/down function (#4524) 2017-06-14 17:34:11 -07:00
splinter98 8293f546af Add support for MQTT server as a notification target (#4474)
This implementation is similar to AMQP notifications:

* Notifications are published on a single topic as a JSON feed
* Topic is configurable, as is the QoS. Uses the paho.mqtt.golang
  library for the mqtt connection, and supports connections over tcp
  and websockets, with optional secure tls support.
* Additionally the minio server configuration has been bumped up
  so mqtt configuration can be added.
* Configuration migration code is added with tests.

MQTT is an ISO standard M2M/IoT messaging protocol and was
originally designed for applications for limited bandwidth
networks. Today it's use is growing in the IoT space.
2017-06-14 17:27:49 -07:00
Anis Elleuch af8071c86a xl: Fix rare freeze after many disk/network errors (#4438)
xl.storageDisks is sometimes passed to some low-level XL functions. Some disks in
xl.storageDisks are set to nil when they encounter some errors. This means all
elements in xl.storageDisks will be nil after some time which lead to an unusable XL.
2017-06-14 17:14:27 -07:00
Daniel Lind dce76d9307 Fix xl.diskWithAllParts to proper checksum algorithm (#4509) 2017-06-14 17:13:02 -07:00
Harshavardhana 11c4223f2c Add docker release files. (#4473)
We used to release by building directly on the docker
hub auto build process, which is sufficient for edge
but it is not a good idea to do it for stable releases.

Do not build docker release binaries again, but instead
use the released binaries themselves which are signed
and validated.
2017-06-14 02:08:35 -07:00
Nitish Tiwari 26903da8c2 Add steps to remove Minio volumes in the swarm (#4536) 2017-06-13 16:10:11 -07:00
Harshavardhana 353f2d3a6e fs: Hold format.json readLock ref to avoid GC. (#4532)
Looks like if we follow pattern such as

```
_ = rlk
```

Go can potentially kick in GC and close the fd when
the reference is lost, only speculation is that
the cause here is `SetFinalizer` which is set on
`os.close()` internally in `os` stdlib.

This is unexpected and unsual endeavour for Go, but
we have to make sure the reference is never lost
and always dies with the server.

Fixes #4530
2017-06-13 08:29:07 -07:00
Nitish Tiwari c8947af227 Update Kubernetes-yaml deployment example and Helm deployment doc with Minio image update steps (#4515) 2017-06-13 01:37:14 -07:00
Harshavardhana 075b8903d7 fs: Add safe locking semantics for format.json (#4523)
This patch also reverts previous changes which were
merged for migration to the newer disk format. We will
be bringing these changes in subsequent releases. But
we wish to add protection in this release such that
future release migrations are protected.

Revert "fs: Migration should handle bucketConfigs as regular objects. (#4482)"
This reverts commit 976870a391.

Revert "fs: Migrate object metadata to objects directory. (#4195)"
This reverts commit 76f4f20609.
2017-06-12 17:40:28 -07:00
Harshavardhana b8463a738c Add support for DCOS host detection, improve Docker detection. (#4525)
isDocker was currently reading from `/proc/cgroup` file. But
this file alone is rather not conclusive evidence. Docker
internally has `.dockerenv` as a special file which we should
use instead.

Fixes #4456
2017-06-13 00:33:21 +00:00
Frank Wessels 6f4862659f Investigate issue #4461 (#4521)
* Code to investigate issue #4461 (rare test failure in TestListenAndServeTLS)

* Use UTCNow() instead of time.Now().UTC()
2017-06-13 00:20:29 +00:00
Nitish Tiwari f59d7a04b4 Removed references to docker-machine in Swarm guide (#4502) 2017-06-10 21:44:20 -07:00
Dee Koder b28d5fa633 Clarify macOS instructions on brew paths. Deleted homebrew upgrade instructions. (#4501) 2017-06-09 14:24:32 -07:00
Harshavardhana 48dbd49980 Add support for kubernetes host detection (#4514)
Additionally improve what we print for `docker pull`
such that its precisely the relevant release tag.

Fixes #4456
2017-06-09 02:42:12 -07:00
Bala FA 3dfe254a11 gateway: make each backend as subcommands. (#4506)
Fixes #4450
2017-06-08 23:28:45 -07:00
Krishna Srinivas ec2920e981 Allow "minio server ." to start minio in fs mode (#4513) 2017-06-08 18:58:51 -07:00
Rushan 0f5483f497 browser: Disable usage/free stats for browser-gateway (#4497) 2017-06-08 15:09:50 -07:00
Krishnan Parthasarathi 8a6b0cc0cd TestInitListeners: Use port 0 pick available port (#4508) 2017-06-08 12:08:21 -07:00
Krishna Srinivas 2c56788f8d Validate gateway arguments (#4376)
Fixes #4355
2017-06-08 11:20:56 -07:00
Frank Wessels 145328ac9f tests: Run select statement in separate goroutine (#4499)
Instead of after the wg.Wait() so as to make sure that the 'earliest'
of the two select case that becomes active is selected..
2017-06-08 07:39:50 -07:00
poornas 45a568dd85 Give more specific error message on browser for nested policies (#4488) 2017-06-07 19:31:23 -07:00
Frank Wessels 7dcc1e92b4 Prevent unnecessary (superfluous) initialization of return variable (#4490) 2017-06-08 00:24:46 +00:00
poornas 999ae1cb96 Fix browser download returning zero bytes for s3 (#4483) 2017-06-06 18:19:35 -07:00
poornas 6651c2fc5f disable settings change on browser in gateway mode (#4472) 2017-06-06 14:56:41 -07:00
Harshavardhana 976870a391 fs: Migration should handle bucketConfigs as regular objects. (#4482)
Current code failed to anticipate the existence of files
which could have been created to corrupt the namespace such
as `policy.json` file created at the bucket top level.

In the current release creating such as file conflicts
with the namespace for future bucket policy operations.
We implemented migration of backend format to avoid situations
such as these.

This PR handles this situation, makes sure that the
erroneous files should have been moved properly.

Fixes #4478
2017-06-06 12:15:35 -07:00
Harshavardhana f99987e47c Generate sha1sum as well for release for backward compatibility. (#4475)
Additionally remove support for arm6vl in release, since
go 1.8 the support for armv6 has been dropped and we do
not see high usage events from this platform.
2017-06-06 11:25:06 -07:00
Harshavardhana 1c3f244fc5 creds: Secretkey should be generated upto 40 characters in length. (#4471)
Current code allowed it wrongly to generate secret key upto 100
we should only use 100 as a value to validate but for generating
it should be 40.

Fixes #4470
2017-06-05 15:18:03 -07:00
Aditya Manthramurthy 986aa8fabf Bypass network in lock requests to local server (#4465)
This makes lock RPCs similar to other RPCs where requests to the local
server bypass the network. Requests to the local lock-subsystem may
bypass the network layer and directly access the locking
data-structures.

This incidentally fixes #4451.
2017-06-05 12:25:04 -07:00
poornas 2559614bfd fix: Set UIversion in reply for policy API (#4469) 2017-06-05 08:11:54 -07:00
Harshavardhana a4d1ef1b62 browser: update ui-assets with new changes. (#4467)
Fixes #4269
2017-06-02 15:11:47 -07:00
Harshavardhana 432bf7d99e Fail if formatting is wrong in our CI tests. (#4459)
We didn't fail before, we should helps in avoiding
formatting issues to creep into the codebase.
2017-06-02 14:05:51 -07:00
poornas 18c4e5d357 Enable browser support for gateway (#4425) 2017-06-01 09:43:20 -07:00
Aditya Manthramurthy 64f4dbc272 Disable redirect of HTTP request to a HTTPS Minio server (#4454)
Fixes #4452
2017-05-31 20:33:13 -07:00
Frank Wessels 9ba57a8df0 Add errCorruptedFormat to list of ignored errors for metadata operations. (#4447)
Fixes listing of objects where xl.json is empty or corrupted to skip to the next disk/server (issue 4354).
2017-05-31 20:03:32 -07:00
Dee Koder 5621e6a494 Refactor service stop signal message. (#4428) 2017-05-31 11:53:04 -07:00
Frank Wessels 0f0758aece Load IO error count for posix atomically (#4448)
* Load error count atomically in order to check for maximum allowed number of IO errors.

* Remove unused (previously atomic) network IO error count
2017-05-31 09:22:53 -07:00
Aditya Manthramurthy a0e02f43e1 Fix and cleanup update message and improve related tests (#4361)
Fixes #4232
2017-05-31 09:22:00 -07:00
Harshavardhana 28352f3f5d log: Startup banner should strip standard ports. (#4443)
APIEndpoints list should strip off standard ports
to avoid confusion with clients.
2017-05-31 09:21:28 -07:00
Harshavardhana 975972d57e server: Redirection should use globalMinioPort with host without port. (#4445)
Currently redirection doesn't work in following scenarios

 - server started with port ":80" and TLS is configured
   client requested insecure request on port "80"
   gets redirected to port 443 and fails.
2017-05-31 09:21:02 -07:00
Harshavardhana 458f22f37c log: Fix printing of signature error request headers. (#4444)
The following commit f44f2e341c
fix was incomplete and we still had presigned URLs printing
in query strings in wrong fashion.

This PR fixes this properly. Avoid double encoding
percent encoded strings such as

`s3%!!(MISSING)A(MISSING)`

Print properly as json encoded.

`s3%3AObjectCreated%3A%2A`
2017-05-31 00:11:06 -07:00
Krishna Srinivas 0bba3cc8e3 gateway-azure: Convert S3 metadata to azure metadata (#4384)
fixes #4292
2017-05-30 20:05:41 -07:00
Cesar Alvernaz bac5303b10 Some minor fixes (#4441)
Word terminations
2017-05-30 11:03:25 -07:00
Harshavardhana e01b2fc06d Disable network share test, appveyor bug. (#4446) 2017-05-30 11:02:31 -07:00
Harshavardhana 072fcf3ba6 fs: Make sure to validate bucket first in PutObject() (#4427)
Currently even when bucket doesn't exist we wrongly
return success, when an object is a directory prefix with
 '/' as suffix and is of size 0.

This PR fixes this behavior.
2017-05-25 09:22:43 -07:00
Harshavardhana b78f6fbcc5 Do not send envVars in ServerInfo() (#4422)
Sending envVars along with access and secret
exposes the entire minio server's sensitive
information. This will be an unexpected
situation for all users.

If at all we need to look for things like if
credentials are set through env, we should
only have access to only this information
not the entire set of system envs.
2017-05-24 21:09:23 -07:00
samkevich 99ca8a2928 fix InvalidAccessKeyId error according to amazon documentation (#4404)
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
2017-05-23 20:07:52 -07:00
poornas 9b3dd44607 Add dotnet library to minio startup message (#4410) 2017-05-23 13:57:27 -07:00
Krishnan Parthasarathi 3c5db69ffd Treat 0.0.0.0 as local address in --address flag (#4386) 2017-05-23 12:07:39 -07:00
Krishna Srinivas 28c26a9e59 Generate random ETag if client does not provide MD5 for PutObjectPart (#4385)
fixes #4289
fixes #4290
2017-05-22 15:48:48 -07:00
morph027 9136734c02 added secrets to distributed swarm minio (#4374) 2017-05-22 11:30:30 -07:00
Harshavardhana 348aa6566c Add nsswitch.conf to our docker image (#4379)
This will cause the alpine images to resolve DNS
using /etc/hosts along with the normal DNS resolver.
2017-05-22 07:05:39 -07:00
luomeiqin 9d98bf1c0f Bucket names can contain hyphen (#4324) 2017-05-19 07:30:00 -07:00
Rushan a767ad321a Browser: Fix Safari Blob download issue (#4357) 2017-05-17 17:03:44 -07:00
Harshavardhana 1886d94e95 server/mux: Use constants provided by Go http (#4360) 2017-05-17 11:57:52 -07:00
Anis Elleuch 542f7ae42c gateway: Reject endpoint pointing to local gateway (#4310)
Show an error when the user enters an endpoint url pointing
to the gateway server itself.
2017-05-16 21:13:29 -07:00
Harshavardhana 59b3e0b79b auth/rpc: Add RWMutex instead of Mutex for granular locking. (#4352)
Refer https://github.com/minio/minio/issues/4345
2017-05-16 14:34:56 -07:00
Aditya Manthramurthy 8975da4e84 Add new ReadFileWithVerify storage-layer API (#4349)
This is an enhancement to the XL/distributed-XL mode. FS mode is
unaffected.

The ReadFileWithVerify storage-layer call is similar to ReadFile with
the additional functionality of performing bit-rot checking. It
accepts additional parameters for a hashing algorithm to use and the
expected hex-encoded hash string.

This patch provides significant performance improvement because:

1. combines the step of reading the file (during
erasure-decoding/reconstruction) with bit-rot verification;

2. limits the number of file-reads; and

3. avoids transferring the file over the network for bit-rot
verification.

ReadFile API is implemented as ReadFileWithVerify with empty hashing
arguments.

Credits to AB and Harsha for the algorithmic improvement.

Fixes #4236.
2017-05-16 14:21:52 -07:00
Frank cae4683971 Make clearing of stale debug lock info independent of deleting map entry of lock itself. (#4353)
This is believed to address issue #4337 where stale information for debug locks in shown.
2017-05-16 07:19:17 -07:00
Krishna Srinivas 5db1e9f3dd signature: use region from Auth header if server's region not configured (#4329) 2017-05-15 18:17:02 -07:00
Anis Elleuch 465274cd21 server-info: Change Error type to string (#4346)
Golang std error type doesn't marshal/unmarshal with json. So errors
are not actually being sent when a client calls ServerInfo() API.
2017-05-15 07:28:47 -07:00
Harshavardhana 87fb911d38 Rename structs for azure and s3 gateway to be consistent. (#4347) 2017-05-15 00:52:33 -07:00
Harshavardhana 155a90403a fs/erasure: Rename meta 'md5Sum' as 'etag'. (#4319)
This PR also does backend format change to 1.0.1
from 1.0.0.  Backward compatible changes are still
kept to read the 'md5Sum' key. But all new objects
will be stored with the same details under 'etag'.

Fixes #4312
2017-05-14 12:05:51 -07:00
Harshavardhana c63afabc9b build/release: Generate sha256sums also without the release tag. (#4318)
Ref #4306
2017-05-12 21:40:22 -07:00
poornas 0404e747cf Fix broken link (#4344) 2017-05-12 18:29:50 -07:00
Aditya Manthramurthy 3bc9e6101c Add minimum requirements sections to notifications docs (#4328) 2017-05-11 17:34:27 -07:00
Anis Elleuch f2ed149714 Add slack channel link to corrupted disk err msg (#4270) 2017-05-11 14:27:32 -07:00
Harshavardhana 5a16dcf4cf Add a graceful msg when CTRL+C is pressed. (#4248) 2017-05-11 14:27:18 -07:00
Harshavardhana fa3d5d0f46 build/release: Generate sha256sums for built binaries. (#4311)
We used to build sha1sum deprecate it and
use sha256sum instead.

Fixes #4306
2017-05-10 11:22:05 -07:00
Krishna Srinivas bb292e4e38 web-handler: Allow anonymous download of zip (#4309)
fixes #4230
2017-05-10 09:54:24 -07:00
poornas d1971b9a4d Prevent duplicate policy rows from being created (#4276) 2017-05-10 09:52:31 -07:00
Harshavardhana fa3f6d75b6 fs: Verify if parent is an object before i/o. (#4304)
PutObject() needs to verify and fail.

Fixes #4301
2017-05-09 17:46:46 -07:00
Harshavardhana 298b470f69 fs/erasure: Ignore objects with / even for DeleteObject() (#4303)
Additionally GetObject() also returns errFileNotFound similar
to HeadObject().

Fixes #4302
2017-05-09 14:32:24 -07:00
Krishna Srinivas fc774957fe gateway: reject requests with unknown authorization (#4297) 2017-05-09 07:53:31 -07:00
Nitish Tiwari c6258f5e97 Multi tenancy doc (#4215)
* Add multi-tenancy doc

* Multi-tenancy documents

* Remove intro

* Update deploy-multiple-minio.md

* Update deploy-multiple-minio.md

* Update deploy-multiple-minio.md

* Update deploy-multiple-minio.md

* Update multi-tenant details

* Remove file

* Rename deploy-multiple-minio.md to README.md

* update ports

* Add multi-tenancy diagrams

* Link diagrams and update disk name in the commands

* Fix tenant config directory
2017-05-08 19:22:34 -07:00
Anis Elleuch 85bc6003e9 gateway-s3: Avoid x2 double quotes in ListParts (#4295)
ListParts response returns doubled double quotes in ETag field.
This commit cleans ETag when receiving it from minio client to
fix the issue.
2017-05-08 14:42:05 -07:00
Nitish Tiwari 0d9de50e21 Bump Docker compose file to latest release (#4271)
* Bump Docker compose file to latest release
* Bump Docker Swarm compose file to latest release
2017-05-07 18:58:24 -07:00
Rushan d13aa1c42d browser: make input number types readonly in share objects modal (#4273) 2017-05-07 18:02:19 -07:00
Harshavardhana 610dbe3479 config: Do not migrate config file if not needed. (#4264)
Also improve the error message returned by `pkg/quick`.

Fixes #4233
2017-05-06 10:16:59 -07:00
Anis Elleuch 2df1e2e9a9 doc: Fix pgsql cmd example (#4265) 2017-05-05 15:20:29 -07:00
Harshavardhana e372b5ed67 build: Fix release build names. (#4263)
Currently due to the occurrence of 6 arguments from
`gen-ldflags.go` leads to a bug where the binaries
genenerated have wrong names.

As shown below.

```
If you want to build for all, Just press Enter: linux/amd64
-->    linux/amd64:github.com/minio/minio
$ ls release/linux-amd64/
[2017-05-04 23:08:51 PDT]  17MiB minio
[2017-05-04 23:08:51 PDT]  17MiB minio.2017-05-05T06:08:22Z
[2017-05-04 23:08:51 PDT]    76B minio.shasum
```

This PR fixes this issue by retaining the previous release
binary names.

```
If you want to build for all, Just press Enter: linux/amd64
-->    linux/amd64:github.com/minio/minio
$ ls release/linux-amd64/
[2017-05-04 23:08:51 PDT]  17MiB minio
[2017-05-04 23:08:51 PDT]  17MiB minio.RELEASE.2017-05-05T06-08-22Z
[2017-05-04 23:08:51 PDT]    76B minio.shasum
```
2017-05-05 13:16:58 -07:00
Harshavardhana 76f4f20609 fs: Migrate object metadata to objects directory. (#4195)
Fixes #3352
2017-05-05 08:49:09 -07:00
Harshavardhana 99ddd35343 docs: use IEC format such as iB everywhere. (#4247) 2017-05-05 08:28:08 -07:00
Remco Verhoef 01e9adc4b3 Implement anonymous uploads, fixes #4250 (#4259) 2017-05-04 20:03:56 -07:00
Nitish Tiwari 4c63fd06c6 Add Kubernetes yaml file deployment example (#4262)
* Add Kubernetes yaml file deployment example

* Change the Docker image tag to latest

* Update latest release tag
2017-05-04 19:30:46 -07:00
Nitish Tiwari bb4efbf258 Add minimum requirements for MySQL notification (#4260) 2017-05-04 17:30:56 -07:00
Harshavardhana a89c7299d1 browser: Update ui-assets with new fixes. (#4246)
Brings two fixes.

 - browser: Listing should append instead of replacing previous listing (#4188)
 - browser: Make login form browser auto-fill compatible (#4091) fixes #4235
 - browser: Selecting a new bucket appends objects list to previous bucket's list (#4252)
2017-05-04 14:57:41 -07:00
Harshavardhana df027a8f51 Webhook endpoints can fail, we must start the server. (#4255)
This PR fixes a regression introduced in #4060
2017-05-04 13:43:54 -07:00
Aditya Manthramurthy a02575ebf9 Bump up minio-go to (fixes #4243) (#4256) 2017-05-04 13:43:23 -07:00
Krishna Srinivas 972a527b66 browser: Selecting a new bucket appends objects list to previous bucket's list (#4252) 2017-05-04 11:12:46 -07:00
Krishnan Parthasarathi 02910725c5 Make gateway help for s3/azure similar (#4249) 2017-05-04 10:38:48 -07:00
Harshavardhana 0ea8bfaf78 Add waiting on hosts in docker entrypoint for distributed setups. (#4244)
Thanks to Remco Verhoef <remco@dutchcoders.io> for the script.

Fixes #4225
2017-05-04 00:48:13 -07:00
Remco Verhoef 069cf9e8aa Use s3.amazonaws.com as default endpoint, fixes #4240 (#4242) 2017-05-03 22:41:03 -07:00
Aditya Manthramurthy 2121b78ea7 Fix bug in JSON representation of object properties (#4238)
Introduced in #4003
2017-05-03 20:10:00 -07:00
Remco Verhoef 5016649f47 Add s3 backend to help, fixes #4219 (#4221)
* Add s3 backend to help, fixes #4219

* Add samples for Gateway usage with S3
2017-05-03 17:55:30 -07:00
Bala FA 2b78444056 fix: ignore TLS handshake error. (#4227)
Fixes #4200
2017-05-03 03:23:15 -07:00
Karthic Rao 9b58a669e5 tests: Fix rare test crash (#4175)
Fix rare test crash by improving the randomness logic.
2017-05-02 23:54:22 -07:00
Krishna Srinivas e5b2e25caf gateway-s3: vendor-update minio-go (#4220) 2017-05-02 18:46:39 -07:00
Krishna Srinivas 4aa65910e5 gateway: Restore bucket policy functionality for Azure (#4209) 2017-05-02 12:27:25 -07:00
Harshavardhana 8b272a3163 config: Improve config migrate messaging. (#4216)
Previous message

```
Migration from version ‘17’ to ‘18’ completed successfully.
```

For example didn't provide any meaningful insights.

This PR attempts to improve this message as below

```
Configuration file '/home/harsha/.minio/config.json' migrated from version '17' to '18' successfully.
```

Fixes #4199
2017-05-02 11:43:27 -07:00
Harshavardhana f0b5c0ec7c windows: Support all REPARSE_POINT attrib files properly. (#4203)
This change adopts the upstream fix in this regard at
https://go-review.googlesource.com/#/c/41834/ for Minio's
purposes.

Go's current os.Stat() lacks support for lot of strange
windows files such as

 - share symlinks on SMB2
 - symlinks on docker nanoserver
 - de-duplicated files on NTFS de-duplicated volume.

This PR attempts to incorporate the change mentioned here

   https://blogs.msdn.microsoft.com/oldnewthing/20100212-00/?p=14963/

The article suggests to use Windows I/O manager to
dereference the symbolic link.

Fixes #4122
2017-05-02 02:35:27 -07:00
Remco Verhoef 44d53c9c67 cleanup and fix comments (#4212) 2017-05-01 14:44:31 -07:00
Krishna Srinivas 6cf6828a4c gateway: Rename gateway files to have "gateway-" prefix (#4207) 2017-05-01 10:32:18 -07:00
Krishna Srinivas 01f04c717e gateway: reject bad path segments in URL (#4202) 2017-04-28 17:17:18 -07:00
Krishna Srinivas 0d32b22359 gateway: Fix help message for gateway (#4201) 2017-04-28 16:42:16 -07:00
Harshavardhana cab298d68f pkg: Update the rpm spec with latest release. (#4187) 2017-04-28 12:35:02 -07:00
Krishna Srinivas 1ea53b4d9f browser: Listing should append instead of replacing previous listing (#4188)
Fixes #4144
2017-04-28 09:30:26 -07:00
Anis Elleuch d36dd80a8a cors: Set Access-Control-Allow-Credentials to true (#4185)
This allow browsers to send credentials with preflighted requests.
2017-04-27 12:40:22 -07:00
Remco Verhoef 3a539ce660 Implement gateway S3 support (#3940) 2017-04-27 11:26:00 -07:00
Harshavardhana 57c5c75611 web: Simplify and converge common functions in web/obj API. (#4179)
RemoveObject() in webAPI currently re-implements some part
of the code to remove objects combine them for simplicity
and code convergence.
2017-04-26 23:27:48 -07:00
Bala FA cf1fc45142 Improve duration humanization. (#4071) 2017-04-26 03:38:35 -07:00
Dee Koder 64c1c0f37d docs: Update with home brew special note for macOS upgrades. (#4180) 2017-04-25 20:18:48 -07:00
Dee Koder 82857cd6df docs: Document homebrew install path changes for minio. (#4178)
* docs: Document homebrew install path changes for minio.

* updates: Updated with the feedback provided.

* docs: Fix breaking change message as per feedback.

* docs: fix typos in readme.

* typo: grammar refactor of update instructions.
2017-04-25 19:43:22 -07:00
Krishnan Parthasarathi 3cdc0c57c8 Provide command to help fill issue template (#4174) 2017-04-25 00:58:11 -07:00
Harshavardhana dc365bca44 build: -s -w should be added by gen-ldflags.go (#4172) 2017-04-24 23:01:38 -07:00
Harshavardhana 3b1626216d docs: Point docker compose to new release. (#4171) 2017-04-24 21:37:13 -07:00
Harshavardhana 48aa2ac392 server: Validate path for bad components in a handler. (#4170) 2017-04-24 18:13:46 -07:00
Frank 0d1e2ab509 Remove hardcoded min and max limit for erasure coding (#4157) 2017-04-24 10:00:33 -07:00
Nitish Tiwari ebf4c447bb docs: Add Minikube deployment to k8s docs (#4133) 2017-04-24 09:20:51 -07:00
Peter Tribble 2b96d9f706 Enable build on solaris (#4115) 2017-04-23 11:10:18 -07:00
Anis Elleuch 83abad0b37 admin: ServerInfo() returns info for each node (#4150)
ServerInfo() will gather information from all nodes before returning
it back to the client.
2017-04-21 07:15:53 -07:00
Harshavardhana df346753e1 api: Fix registering of s3 endpoint peers properly (#4159)
We need to have local peer initialized properly
for listen bucket to work, current code did initialize
properly but the resulting code was initializing
peer on a wrong target v/s what listen bucket expected
it to be.

This regression came in de204a0a52

Fixes #4158
2017-04-20 15:28:29 -07:00
Harshavardhana f1d7780167 lock: Vendorize all the new changes made in minio/dsync (#4154)
Fixes #4139
2017-04-19 14:22:35 -07:00
Harshavardhana 5a3c5aec31 server/mux: Fix serverMux to set deadlines based on UTC time. (#4146)
Avoid using `time.Now()` instead rely on UTC time
for the final deadline, this is to be consistent with
all our internal functions.

Reduce the default read timeout to 15 seconds
in lieu with a newly discovered issue
   - https://github.com/minio/minio/issues/4139

Additionally also change the Read() conn wrapper
to set deadline only upon successful Reads().
2017-04-19 13:16:06 -07:00
Aditya Manthramurthy a4305742e8 Add key for Kafka messages (fixes #4143) (#4151) 2017-04-19 11:26:35 -07:00
Harshavardhana 640ebb2f79 lock: Fix missing formatting directives while printing. (#4147)
Current log prints in this form

```
ERRO[8150] Lock maintenance failed to remove entry for write
lock (should never happen)%!!(MISSING)(EXTRA ....
```

Fix this by using proper formatting directive.
2017-04-19 10:37:56 -07:00
Harshavardhana 402a5e3bea docs: Fix and reword FreeBSD documentation. (#4145) 2017-04-19 00:25:20 -07:00
Harshavardhana f4dac979a2 server: Fix message when corrupted or unsupported format is found. (#4142)
Refer https://github.com/minio/minio/issues/4140

This is a fix to provide a little more elaborate message.
2017-04-18 10:35:17 -07:00
Krishnan Parthasarathi 3032f0f505 Remove duration field from lock instrumentation (#4111)
Duration for which a lock was held can be computed from the `Since`
field of `OpsLockState`. It is the difference between current time and
time at which the namespace lock was held. This change avoids
superfluous instrumentation.
2017-04-15 11:40:01 -07:00
Harshavardhana 7765081db7 cache: Increasing caching GC percent from 20 to 50. (#4041)
Previous value was set to avoid large cache value build
up but we can clearly see this can cause lots of GC
pauses which can lead to significant drop in performance.

Change this value to 50% and decrease the value to 25%
once the 75% cache size is used. To have a larger
window for GC pauses.

Another change is to only allow caching if a server has
more than 24GB of RAM instead of 8GB.
2017-04-15 02:16:49 -07:00
Anabel V 18bfe5cba6 docs: Created new illustration for docs. (#4012) 2017-04-14 17:21:57 -07:00
Anis Elleuch 14f0047295 fs: Remove fs meta lock when PutObject() fails (#4114)
Removing the fs meta lock file when PutObject() encounters any error
during its execution, such as upload getting permatuerly cancelled
by the client.
2017-04-14 12:06:24 -07:00
Krishna Srinivas e6b2253da9 gateway: Fix help message for custom Azure Blob Storage endpoint. (#4113) 2017-04-14 11:02:43 -07:00
Krishnan Parthasarathi ca64b86112 Return possible states a heal operation (#4045) 2017-04-14 10:28:35 -07:00
Karthic Rao 5f065e2a96 server: Fix CI build complaints (#4119)
- Ineffassign fixes.
- Spell check correction.
2017-04-14 08:00:04 -07:00
Harshavardhana a7afa469e2 xl: Add stat calls to keep track of ignored errors. (#4117)
Such that in a situation where all errors were
ignored we need to reduce the errors using
readQuorum to get a consistent error value.

Without this change errors generated will
never be consistent with for an expected scenario.

For example in a 6 disk setup 1 disk is missing
and 5 do not have the volume (testbucket)

Without this change Stat() would result in different
errors depending on which disk died. Can cause
confusion to S3 client application.

This change addresses need to track type of
errors we ignored and bring readQuorum to
choose the maximally occuring as the value
of truth.
2017-04-14 01:46:16 -07:00
Bala FA d103d5fb7c server: Error out if loopback addr is used for Distributed Erasure (#4105) 2017-04-12 20:27:24 -07:00
Harshavardhana 6683247080 tests: Fix the sopradic test failure in TestListObjectPartsDiskNotFound (#4107)
getBucketInfo() should keep track errors ignored,
such that in a situation where all errors were
ignored we need to reduce the errors using readQuorum
to get a consistent error value.

This is the problem we see with DiskNotFound test
disks are randomly removed.

Fixes #4095
2017-04-12 15:38:35 -07:00
Anis Elleuch e4bd882f11 handlers: Ignore malformatted datetime type header (#4097)
Ignore headers, such as If-Modified-Since, If-Unmodified-Since, etc.. when they
are received with a format other than HTTP date.
2017-04-12 12:34:57 -07:00
Nitish Tiwari 4448285a83 Cleanup service docs (#4103)
- Remove windows service doc and add to Minio-service
- Remove Linux service doc as it is duplicated
2017-04-12 12:18:34 -07:00
Harshavardhana 952c618441 server: Fix a regression in printing startup banner. (#4100)
Octect based sorting was lost in the previous commit

de204a0a52

This PR fixes a regression - fixes #4099
2017-04-12 09:22:35 -07:00
Krishna Srinivas c5249c35d3 gateway: Support for custom endpoint. (#4086) 2017-04-11 17:44:26 -07:00
Bala FA de204a0a52 Add extensive endpoints validation (#4019) 2017-04-11 15:44:27 -07:00
Harshavardhana 1b1b9e4801 lock/rpc: change rpcPath to be called serviceEndpoint. (#4088)
This is a cleanup to ensure proper naming.
2017-04-11 10:25:21 -07:00
Rushan 1b0b2c1c76 Browser: Make login form browser auto-fill compatible (#4091) 2017-04-11 10:11:42 -07:00
koolhead17 7f5e037846 docs: Update docker quick-start guide and adds relevant project URLS (#4075) 2017-04-11 00:53:33 -07:00
Karthic Rao 929a13f33f Fix for writes from Apache Spark. (#4074)
- Due to usage of amazon SDK, spark expects md5sum of empty string to be
  returned when it does PUT on a directory.
- The fix returns md5sum of a empty string for the above mentioned case.
- This fixes the issue of Apache Spark not being able to write into Minio.
2017-04-10 19:51:23 -07:00
Krishna Srinivas a4209c10ea signature-v4: Use sha256("") for calculating canonical request (#4064) 2017-04-10 09:58:08 -07:00
Harshavardhana b927523223 server: Introduce a new env MINIO_REGION. (#4078)
This is implemented to be able to override region
through command line just like how access and
secret keys are provided.
2017-04-09 10:44:10 -07:00
Aditya Manthramurthy 604417baf4 Allow cluster to start when only n/2 servers are up (#4066)
Fixes #3234.

Relaxes the quorum requirement to start the object layer, and skips
quick-healing at start-up (as no write quorum is present).
2017-04-09 00:28:27 -07:00
Harshavardhana 6e9ac8db59 docker: Support docker swarm secrets. (#3977)
Fixes #3896
2017-04-08 01:43:40 -07:00
Harshavardhana 0497d5c342 api: SourceInfo should be populated in GET/HEAD notification. (#4073)
Refer https://github.com/minio/mc/issues/2073
2017-04-08 01:39:20 -07:00
Harshavardhana 6b4f368dfe notify: Webhook endpoints can fail, but we must start the server. (#4060)
Ignore any network errors when registering a webhook
notifier during Minio startup sequence. This way server
can be started even if the webhook endpoint is not available
and unreachable.
2017-04-08 01:13:55 -07:00
Nitish Tiwari 6507c30bbc Add steps to run Minio distributed on Windows (#4068) 2017-04-08 01:12:00 -07:00
Harshavardhana f44f2e341c log: Dump signature request properly. (#4063)
Currently percent encoded strings are not properly encoded.

`s3%!!(MISSING)A(MISSING)`

Print properly as json encoded.

`s3%3AObjectCreated%3A%2A`
2017-04-07 14:37:32 -07:00
Krishnan Parthasarathi 60ea2b17ba Fix xml block syntax in admin-api Readme (#4062) 2017-04-07 03:38:01 -07:00
Harshavardhana 5ed1a8ad23 docs: macOS brew now refers to Minio fork (#4059) 2017-04-07 01:42:59 -07:00
Harshavardhana 27749c2124 admin/info: Add HTTPStats value as part of serverInfo() struct. (#4049)
Remove our counter implementation instead use atomic external
package which supports more types and methods.
2017-04-06 23:08:33 -07:00
Krishna Srinivas 1d99a560e3 refactor: extractSignedHeaders() handles headers removed by Go http server (#4054)
* refactor: extractSignedHeaders() handles headers removed by Go http server.
* Cleanup extractSignedHeaders() TestExtractSignedHeaders()
2017-04-05 17:00:24 -07:00
Krishna Srinivas af82d27018 signature-v4: Support for transfer-encoding request header (#4053) 2017-04-05 15:08:33 -07:00
Anis Elleuch f205689ff5 build: Fix compilation in 32 bits platforms (#4052)
go fails to build Minio under at least, armv6 and 386 due to some
inconsistencies in the type of one syscall variable in different
architectures. This PR casts that variable to uint64 to achieve
the desired consistency.
2017-04-05 11:17:59 -07:00
Harshavardhana 393c01d078 browser: Generate new UI assets. 2017-04-04 23:03:36 -07:00
Rajeesh C V e9037b9d36 fix: add white space in storage usage section (#4038)
Fix for #3962 - add whitespace before storage usage details section in
 Browse component
2017-04-04 09:14:45 -07:00
Harshavardhana 4747adfcb4 fs: Enable returning ETag along with ListObjects() (#4042)
This is to comply with S3 behavior, we previously removed
reading `fs.json` for optimization reasons but we have a
reason to believe that providing ETag and using gjson
provides needed benefit of not having to deal with
unmarshalling overhead of golang stdlib.

Fixes #4028
2017-04-04 09:14:03 -07:00
Anis Elleuch 52d8f564bf sigv2: Unespace canonicalized resources values (#4034)
Values of canonicalized query resources should be unescaped before calculating
the signature. This bug is not noticed before because partNumber and uploadID
values in Minio doesn't have characters that need to be escaped.
2017-04-03 17:55:14 -07:00
Harshavardhana 3fe33e7b15 handler: simplify parsing valid location constraint. (#4040)
Separate out validating v/s parsing logic in
isValidLocationConstraint() into parseLocationConstraint()
and isValidLocation()

Additionally also set `X-Amz-Bucket-Region` as part of the
common headers for the clients to fallback on in-case of any
region related errors.
2017-04-03 14:50:09 -07:00
Krishnan Parthasarathi 4041e5f20d Provide mc-admin-heal command on start-up (#4031)
Healing of buckets, objects and incomplete uploads are implemented and
available via admin REST APIs. Additionally, it is available via mc admin 
sub-command. The warning is no longer relevant.

Fixes #4030
2017-04-03 14:24:25 -07:00
Krishnan Parthasarathi 96c46c15e7 madmin: Rename HealObjectResult to HealResult (#4035)
madmin.HealObjectResult is used in HealObject and HealUpload. It only
makes sense to rename it to HealResult.
2017-04-03 08:25:32 -07:00
Harshavardhana 3bf67668b6 sys/stats: return cgroup mem limit, fall back to sysinfo() (#4002)
This is necessary where in certain environments where
cgroup is used to limit memory usage of a container or
a particular process.

GetStats() is used by caching module to figure out the
optimal cacheable size in memory with cgroup limits
what sysinfo reports might not be the right value set
for a given process.

Fixes #4001
2017-04-02 10:46:16 -07:00
Anis Elleuch e31e2c3bc2 doc: Explain how to create certificate chain file (#4032)
public.crt needs sometimes to have a chain certificate, this PR
explains how to construct public.crt when certificate are issued
by a certificate authority.
2017-04-02 04:47:56 -07:00
Harshavardhana 214279aa57 build: Reduce binary size by using -s -w (#4027)
Refer #3939
2017-04-01 01:06:16 -07:00
Harshavardhana 4de6b15fca vet: Fix all the go vet complaints (#4029)
```
go tool vet -atomic -bool -copylocks -nilfunc \
   -printf -shadow -rangeloops -unreachable \
   -unsafeptr -unusedresult cmd/
```
2017-04-01 01:06:06 -07:00
Krishnan Parthasarathi 2bd694dbc8 Add disksUnavailable healStatus const (#3990)
`disksUnavailable` healStatus constant indicates that a given object
needs healing but one or more of disks requiring heal are offline. This
can be used by admin heal API consumers to distinguish between a
successful heal and a no-op since the outdated disks were offline.
2017-03-31 17:55:15 -07:00
Aditya Manthramurthy a2a8d54bb6 Add access format support for Elasticsearch notification target (#4006)
This change adds `access` format support for notifications to a
Elasticsearch server, and it refactors `namespace` format support.

In the case of `access` format, for each event in Minio, a JSON
document is inserted into Elasticsearch with its timestamp set to the
event's timestamp, and with the ID generated automatically by
elasticsearch. No events are modified or deleted in this mode.

In the case of `namespace` format, for each event in Minio, a JSON
document is keyed together by the bucket and object name is updated in
Elasticsearch. In the case of an object being created or over-written
in Minio, a new document or an existing document is inserted into the
Elasticsearch index. If an object is deleted in Minio, the
corresponding document is deleted from the Elasticsearch index.

Additionally, this change upgrades Elasticsearch support to the 5.x
series. This is a breaking change, and users of previous elasticsearch
versions should upgrade.

Also updates documentation on Elasticsearch notification target usage
and has a link to an elasticsearch upgrade guide.

This is the last patch that finally resolves #3928.
2017-03-31 14:11:27 -07:00
Harshavardhana 2040d32ef8 server/tls: Do not rely on a specific cipher suite (#4021)
Do not rely on a specific cipher suite instead let the
go choose the type of cipher needed, if the connection
is coming from clients which do not support forward
secrecy let the go tls handle this automatically based
on tls1.2 specifications.

Fixes #4017
2017-03-31 13:28:45 -07:00
Harshavardhana f1015a5096 notifiers: Stop using url.Parse in validating address format. (#4011)
url.Parse() wrongly parses an address of format "address:port"
which is fixed in go1.8.  This inculcates a breaking change
on our end. We should fix this wrong usage everywhere so that
migrating to go1.8 eventually becomes smoother.
2017-03-31 04:47:40 -07:00
Aditya Manthramurthy 096427f973 Add deliveryMode parameter for AMQP notfication target (#4008)
Configuration migration was done.

Also adds documentation about AMQP configuration parameters.

Fixes #3982
2017-03-31 03:34:26 -07:00
Rushan 5cec6bd80d Browser: Use object name with prefix to delete sub-path objects (#4013) 2017-03-30 23:28:28 -07:00
Bala FA 6e9c91f43a fix: use its own lock in serverConfigV17 (#4014)
Previously serverConfigV17 used a global lock that made any instance of
serverConfigV17 depended on single global serverConfigMu.

This patch fixes by having individual lock per instances.
2017-03-30 22:26:24 -07:00
Bala FA 2df8160f6a server: handle command line and env variables at one place. (#3975) 2017-03-30 11:21:19 -07:00
Romain Bouyé 447fdd4097 docs: Fix typo in docs/config/README.md (#4009) 2017-03-30 11:09:15 -07:00
Harshavardhana 28c5a887de event: Set contentType as well under NotificationEvent. (#4003)
This is an enhancement change to to cater support all
the data fields present on the object. Currently
we only send a subset of data which object info
provides us.

It also helps us keep a full namespace mirror on
notification targets for efficient query.
2017-03-30 08:58:14 -07:00
Anis Elleuch fbe8b3259d webhook: Add support of custom CAs (#4000) 2017-03-29 13:42:55 -07:00
Anis Elleuch e2aba9196f obj-handlers: Rewrite src & dst path cmp in Copy() (#3998)
CopyObjectHandler() was incorrectly performing comparison
between destination and source object paths, which sometimes
leads to a lock race. This PR simplifies comparaison and add
one test case.
2017-03-29 09:21:38 -07:00
Aditya Manthramurthy 61b08137b0 Add access format support for Redis notification target (#3989)
This change adds `access` format support for notifications to a Redis
server, and it refactors `namespace` format support.

In the case of `access` format, a list is used to store Minio
operations in Redis. Each entry in the list is a JSON encoded list of
two items - the first is the Minio server timestamp of the event, and
the second is an object describing the operation that created/replaced
the object in the server.

In the case of `namespace` format, a hash is used. Entries in the hash
may be updated or removed if objects in Minio are updated or deleted
respectively. The field values in the Redis hash are JSON encoded.

Also updates documentation on Redis notification target usage.

Towards resolving #3928
2017-03-29 08:55:53 -07:00
Harshavardhana 1caad902cb config/path: Figure out absolute paths properly on windows. (#3996)
The following form of arguments such as

```
minio.exe -C some_dir server dir
```

has stopped working because of lack of handling of
absolute paths for config directory. Always calculate
absolute path for any relative paths on any operating
system.

The following fix converts all config directory relative
paths into absolute paths.

Fixes #3991
2017-03-29 08:55:33 -07:00
Nitish Tiwari d99efa2c93 Update filename (#3995) 2017-03-28 23:14:28 -07:00
Krishna Srinivas 9ee83b89bb config: Appropriate error message when newer config file is found (#3972) 2017-03-28 18:41:16 -07:00
Nitish Tiwari a8cb43926a Docker guide fix (#3992)
* Update Docker quick start guide

- Add Compose to orchestration section.

- Refer Swarm from Docker quick start guide.

- Add common Docker commands to quick start guide.

* Paragraph cleanup
2017-03-28 13:54:19 -07:00
Harshavardhana b62cd8ed84 sign/streaming: Content-Encoding is not set in newer aws-java-sdks (#3986)
We can't use Content-Encoding to verify if `aws-chunked` is set
or not. Just use 'streaming' signature header instead.

While this is considered mandatory, on the contrary aws-sdk-java
doesn't set this value

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

```
Set the value to aws-chunked.
```

We will relax it and behave appropriately. Also this PR supports
saving custom encoding after trimming off the `aws-chunked`
parameter.

Fixes #3983
2017-03-27 17:02:04 -07:00
Anis Elleuch 1b3a517683 lock, Windows: O_CREAT infers only GENERIC_WRITE (#3981)
Avoid using GENERIC_ALL flag when creating a file since it could
return permission denied in some Windows versions.
2017-03-27 12:47:39 -07:00
Nitish Tiwari a9c0f1e0a4 Added kernel tuning docs (#3921) 2017-03-27 11:29:04 -07:00
Aditya Manthramurthy a099319e66 Support access format for database notification targets (#3953)
* Add configuration parameter "format" for db targets and perform
  configuration migration.
* Add PostgreSQL `access` format: This causes Minio to append all events
  to the configured table. Prefix, suffix and event filters continue
  to be supported for this mode too.
* Update documentation for PostgreSQL notification target.
* Add MySQL `access` format: It is very similar to the same format for
  PostgreSQL.
* Update MySQL notification documentation.
2017-03-27 11:27:25 -07:00
Bala FA 6e63904048 browser-flag: wrapped bool type denotes browser on/off flag. (#3963)
Statically typed BrowserFlag prevents any arbitrary string value
usage. The wrapped bool marshals/unmarshals JSON according to the
typed value ie string value "on" represents boolean true and "off" as
boolean false.
2017-03-26 12:00:27 -07:00
Harshavardhana 565ac4c861 tests: use url.QueryEscape() when dealing with url query params. (#3974)
This is to keep the portability and also avoid errors that
might occur using the functions written for URL resource name
Since query param values have different escaping requirements.
2017-03-26 11:56:17 -07:00
Harshavardhana a4ecd8bca2 docs: Add config directory documentation/guide. (#3889) 2017-03-25 02:34:04 -07:00
Harshavardhana 28eff0f6c1 build: Improve build messaging, say where we built Minio. (#3973) 2017-03-25 00:33:57 -07:00
Krishnan Parthasarathi c27ece409b heal: Check if all parts are available and valid (#3967)
In the algorithm to check if an object requires healing, in addition to
checking if all disks have xl.json present we should check if all parts
of the object are present and have valid blake2b checksums.

Also fixed a minor compilation error in heal-objects-list.go.
2017-03-24 08:40:44 -07:00
Dee Koder 8c0ce2fee9 docs: Removed space from code blocks. (#3965) 2017-03-23 19:06:31 -07:00
Bala FA d3cb79a57c Refactor logger (#3924)
This patch fixes below

* Previously fatalIf() never writes log other than first logging target.
* quiet flag is not honored to show progress messages other than startup messages.
* Removes console package usage for progress messages.
2017-03-23 16:36:00 -07:00
Anis Elleuch 11e15f9b4c config: Remove level in console/file loggers (#3938)
Also rename fileName field in file logger to filename
2017-03-23 08:27:22 -07:00
Krishnan Parthasarathi 4e92b2ecb8 Fix listDirHealFactory merging of entries across disks (#3959)
For listing of objects needing heal, we list all objects present on all
the disks and return the set union. We were incorrectly dropping objects
that weren't already seen in disks so far.

Sample directory layout of disks in a 4-disk setup:
`/tmp/1`, `/tmp/2`, `/tmp/3`, `/tmp/4` are directories used as disks here.
`test` is the bucket, `obj1` and obj2` are the objects.
```
/tmp/1/test
└── obj2
    ├── part.1
    ├── part.2
    └── xl.json
/tmp/2/test
└── obj1
    ├── part.1
    ├── part.2
    └── xl.json
/tmp/3/test
├── obj1
│   ├── part.1
│   ├── part.2
│   └── xl.json
└── obj2
    ├── part.1
    ├── part.2
    └── xl.json
/tmp/4/test
[This is empty]

```
2017-03-23 08:24:59 -07:00
koolhead17 80b83a51a3 Docs: Fix for Self signed certificate. (#3957) 2017-03-23 08:20:39 -07:00
Krishna Srinivas 777d12d928 browser: update ui-assets.go (#3956) 2017-03-22 23:17:14 -07:00
Krishnan Parthasarathi 607c8a9611 Add sourceInfo to NotificationEvent (#3937)
This change adds information like host, port and user-agent of the
client whose request triggered an event notification.

E.g, if someone uploads an object to a bucket using mc. If notifications
were configured on that bucket, the host, port and user-agent of mc
would be sent as part of event notification data.

Sample output:
```
"source": {
          "host": "127.0.0.1",
          "port": "55808",
          "userAgent": "Minio (linux; amd64) minio-go/2.0.4 mc ..."
}
```
2017-03-22 18:44:35 -07:00
Anis Elleuch 7f5a5b5e9d config: Do not validate creds when set via env (#3955)
It is useless to validate access/secret keys stored in
config file when the user sets them in the environment.
2017-03-22 16:11:58 -07:00
Bala FA d4ca2ee1a3 pkg/quick: add Save() function and other enhancements. (#3951)
* Add a new function Save() which saves given configuration into given file.
* Simplify Load() function.
* Remove unused CheckVersion().
* CheckData() is a private function now.
* quick_test.go is part of quick package now.
* minio server uses top level quick.Load() and quick.Save() functions.
2017-03-22 10:23:25 -07:00
Krishnan Parthasarathi 417ec0df56 HealObject should succeed when only N/2 disks have data (#3952) 2017-03-22 10:15:16 -07:00
Rushan fbfb4fc5a0 Browser: Use polyfill to support Object.assign in IE11 (#3942) 2017-03-21 13:48:07 -07:00
koolhead17 97dc34fe06 docs: Fix for README.md markdown table. (#3948) 2017-03-21 12:02:39 -07:00
Krishnan Parthasarathi 9f9ba1e984 XL: Return the right error (#3944) 2017-03-21 10:33:25 -07:00
Krishnan Parthasarathi 13c4ce3617 Add notification for object access via GET/HEAD (#3941)
The following notification event types are available for all targets,
      s3:ObjectAccessed:Get
      s3:ObjectAccessed:Head
      s3:ObjectAccessed:*
2017-03-21 10:32:17 -07:00
Krishnan Parthasarathi 181e002c56 pkg/madmin: Set UploadID in ListUploadsHeal (#3945)
Without this fix, `mc admin heal -I` wouldn't be able to heal ongoing
uploads. `mc` depends on `ListUploadsHeal` API to identify ongoing
uploads to heal given a bucket and an object.
2017-03-21 10:32:02 -07:00
Anis Elleuch 9d6e226692 heal: Set truncate when no more walk entries (#3932) 2017-03-20 15:31:25 -07:00
Krishnan Parthasarathi eb02261642 XL: Don't return ignored errors in listDirFactory (#3935)
Previously, erasure backend's `listDirFactory` may return errors which
were explicitly ignored. With this change, it returns nil. Superfluous
checks at higher-layers for ignored errors are removed as well.
2017-03-20 11:09:05 -07:00
Pawan Rawal 1396e91dd1 Update link for downloading minio server. (#3934)
This and the link for downloading Minio server at other places in the docs seems to be broken. I suppose this happened while updating the name of the page (which updated the url) in Doctor docs. 
Might be nice for Doctor to update internal links if the name of a page is changed in a background job.
2017-03-19 18:26:23 -07:00
Bala FA 7ebf11b202 words: new package Damerau Levenshtein distance function. (#3929) 2017-03-19 14:23:05 -07:00
Bala FA 1c97dcb10a Add UTCNow() function. (#3931)
This patch adds UTCNow() function which returns current UTC time.

This is equivalent of UTCNow() == time.Now().UTC()
2017-03-18 11:28:41 -07:00
Anis Elleuch 3a6111eff5 admin: Export HealStatus data type (#3930)
`healStatus` can be returned to the API caller. This commit will help
developers to declare a variable with HealStatus type.
2017-03-18 11:27:27 -07:00
Aditya Manthramurthy 2463ae243a Add support for MySQL notifications (fixes #3818) (#3907)
As a new configuration parameter is added, configuration version is
bumped up from 14 to 15.

The MySQL target's behaviour is identical to the PostgreSQL: rows are
deleted from the MySQL table on delete-object events, and are
created/updated on create/over-write events.
2017-03-17 09:29:17 -07:00
Krishnan Parthasarathi c192e5c9b2 Implement heal-upload admin API (#3914)
This API is meant for administrative tools like mc-admin to heal an
ongoing multipart upload on a Minio server.  N B This set of admin
APIs apply only for Minio servers.

`github.com/minio/minio/pkg/madmin` provides a go SDK for this (and
other admin) operations.  Specifically,

  func HealUpload(bucket, object, uploadID string, dryRun bool) error

Sample admin API request:
POST
/?heal&bucket=mybucket&object=myobject&upload-id=myuploadID&dry-run
- Header(s): ["x-minio-operation"] = "upload"

Notes:
- bucket, object and upload-id are mandatory query parameters
- if dry-run is set, API returns success if all parameters passed are
  valid.
2017-03-17 09:25:49 -07:00
Nitish Tiwari d4eea224d4 Remove white spaces (#3922) 2017-03-17 09:23:22 -07:00
Nitish Tiwari e55421ebdd Fixed Docker compose link (#3920) 2017-03-17 01:06:11 -07:00
Dee Koder 0a5d57a91e docs: Update gateway doc with roadmap section. (#3918) 2017-03-16 18:25:01 -07:00
Krishna Srinivas cea4cfa3a8 Implement S3 Gateway to third party cloud storage providers. (#3756)
Currently supported backend is Azure Blob Storage.

```
export MINIO_ACCESS_KEY=azureaccountname
export MINIO_SECRET_KEY=azureaccountkey
minio gateway azure
```
2017-03-16 12:21:58 -07:00
Anis Elleuch 8426cf9aec config: Accept more address format + unit test (#3915)
checkURL() is a generic function to check if a passed address
is valid. This commit adds support for addresses like `m1`
and `172.16.3.1` which is needed in MySQL and NATS. This commit
also adds tests.
2017-03-16 11:44:01 -07:00
Harshavardhana f3334159a4 config: Relax browser and region to be empty. (#3912)
- If browser field is missing or empty
  then default to "on".

- If region field is empty or missing then
  default to "us-east-1" (S3 spec behavior)
2017-03-16 11:06:17 -07:00
Harshavardhana 3edff1501e Vendor upstream redis library instead of our fork. (#3913)
We forked the upstream to address a build issue on
go 1.6 - that is long done and we don't need
to manage our forks anymore.
2017-03-16 08:22:47 -07:00
Bala FA 21d73a3eef Simplify credential usage. (#3893) 2017-03-16 00:16:06 -07:00
Krishnan Parthasarathi 051f9bb5c6 Implement list uploads heal admin API (#3885) 2017-03-16 00:15:06 -07:00
Harshavardhana 6509589adb Use canonicalETag helper wherever needed. (#3910) 2017-03-15 20:48:49 -07:00
Anis Elleuch ae4361cc45 config: Check for duplicated entries in all scopes (#3872)
Validate Minio config by checking if there is double json key
in any scope level. The returned error contains the json path
to the duplicated key.
2017-03-15 16:30:34 -07:00
Krishna Srinivas cad0d0eb7a browser: Update ui-assets.go (#3902)
fixes #3898
2017-03-15 11:32:51 -07:00
Nitish Tiwari ba0c11757e Added that no special config changes reqd (#3906)
Added a line saying no special config changes are required for Shared mode. Also the previous `Why Shared Backend` and current `Use cases` are already merged. This fixes the comment: https://github.com/minio/minio/pull/3888#discussion_r105573494
2017-03-15 08:22:03 -07:00
Krishna Srinivas 96050c1e21 browser: Do not show "Loading..." if there are no buckets. (#3904) 2017-03-14 19:07:23 -07:00
Rushan a27d1b3d86 Browser: Use babel-polyfill to support new ES6 built-ins in older browsers (#3900) 2017-03-14 15:02:20 -07:00
Anis Elleuch a5e60706a2 xl,fs: Return 404 if object ends with a separator (#3897)
HEAD Object for FS and XL was returning invalid object name when
an object name has a trailing slash separator, this PR changes the
behavior and will always return 404 object not found, this guarantees
a better compatibility with S3 spec.
2017-03-13 22:20:46 -07:00
Harshavardhana 5f7565762e api: postPolicy cleanup. Simplify the code and re-use. (#3890)
This change is cleanup of the postPolicyHandler code
primarily to address the flow and also converting
certain critical parts into self contained functions.
2017-03-13 14:41:13 -07:00
Nitish Tiwari 3314501f19 Simplify shared mode document (#3888) 2017-03-12 16:17:03 -07:00
Harshavardhana 9d53a646a1 Simplify the title for orchestration and some words. (#3887) 2017-03-11 02:17:03 -08:00
Nitish Tiwari 5e0032e165 Update links (#3886) 2017-03-11 00:11:07 -08:00
Harshavardhana 305952d734 browser: Update ui-assets with new changes. 2017-03-10 15:13:26 -08:00
Rushan e77885d671 Browser: Add Object.assign polyfill to support older browsers (#3884) 2017-03-10 14:30:23 -08:00
Nitish Tiwari 2410eb281e Update kafkacat command with consumer flag (#3882) 2017-03-10 09:01:59 -08:00
Harshavardhana e54025805f browser: update ui-assets with new changes. 2017-03-09 15:26:25 -08:00
Anis Elleuch d602495600 madmin: Do not require SSL to set credentials (#3879)
We need to relax this requirement and let the client decides
if it can allow to set credentials API over plain connection.
2017-03-09 14:08:33 -08:00
Nitish Tiwari 03937e7554 Added documentation for orchestration platforms (#3684) 2017-03-09 14:06:51 -08:00
Harshavardhana e3b627a192 docker: Add ARM64 image build support (#3876) 2017-03-09 13:57:27 -08:00
Rushan ccc3349f0c Browser: Show complete bucket name by removing the ellipsis cut off (#3877) 2017-03-09 10:42:38 -08:00
Harshavardhana 85cbd875fc cleanup: All conditionals simplified under pkg. (#3875)
Address all the changes reported/recommended by
`gosimple` tool.
2017-03-09 10:13:30 -08:00
Harshavardhana 43317530d5 Fix odd shadowing bug in XL init. (#3874)
Fixes #3873
2017-03-08 20:42:45 -08:00
Bala FA 8a9852220d Make unit testable cert parsing functions. (#3863) 2017-03-08 19:20:01 -08:00
Harshavardhana 47ac410ab0 Code cleanup - simplify server side code. (#3870)
Fix all the issues reported by `gosimple` tool.
2017-03-08 10:00:47 -08:00
Harshavardhana 433225ab0d docker: ca-certificates should not be removed. (#3868) 2017-03-07 16:30:19 -08:00
Harshavardhana 3e655a2c85 docker: Add ARM docker container dockerfile. (#3574) 2017-03-07 15:15:05 -08:00
Anis Elleuch a2eae54d11 xl: Respect min. space by checking PrepareFile err (#3867)
It was possible to upload a big file which overcomes the minimal
disk space limit in XL, PrepareFile was actually checking for disk
space but we weren't checking its returned error. This patch fixes
this behavior.
2017-03-07 14:48:56 -08:00
Anis Elleuch 79e0b9e69a Relax minio server start when disk threshold is reached and adds space check in FS (#3865)
* fs: Rename tempObjPath variable in fsCreateFile()
* fs/posix: Factor checkDiskFree() function
* fs: Add disk free check in fsCreateFile()
* posix: Move free disk check to createFile()
* xl: Relax free disk check in POSIX initialization
* fs: checkDiskFree checks for space to store data
2017-03-07 12:25:40 -08:00
Krishna Srinivas 29ff9674a0 browser: Humanize expiry time for Share-Object. (#3861) 2017-03-06 20:13:52 -08:00
Bala FA bff4d29415 Remove commands and commandsTree global variables. (#3855) 2017-03-06 19:35:26 -08:00
Krishna Srinivas 436db49bf3 Share object expiry value modification modal was not working (#3860) 2017-03-06 18:50:25 -08:00
Rushan 966818955e Browser: Implement multiple object delete (#3859) 2017-03-06 15:43:43 -08:00
Harshavardhana e49efcb9d9 xl: quickHeal heal bucket only when needed. (#3854)
This improves the startup time significantly
for clusters which have lot of buckets.

Also fixes a bug where `.minio.sys` is created
on disks which do not have `format.json`
2017-03-06 02:00:15 -08:00
Harshavardhana 6f931d29c4 rpm: Add RPM spec for minio build. (#3853)
Currently the package is built and hosted at

https://copr.fedorainfracloud.org/coprs/minio/minio/

To enable minio repo one has to download.

Fedora - 25
https://copr.fedorainfracloud.org/coprs/minio/minio/repo/fedora-25/minio-minio-fedora-25.repo

Fedora - 26
https://copr.fedorainfracloud.org/coprs/minio/minio/repo/fedora-26/minio-minio-fedora-26.repo

Enables for both i386 and x86_64.

Fixes #3576
2017-03-05 13:09:31 -08:00
Krishnan Parthasarathi e3fd4c0dd6 XL: Make listOnlineDisks and outDatedDisks consistent w/ each other. (#3808) 2017-03-04 14:53:28 -08:00
Harshavardhana b05c1c11d4 browser: Update UI assets with new changes. 2017-03-03 18:05:02 -08:00
Krishna Srinivas 0bae3330e8 browser: Send correct arguments for RemoveObjects web handler. (#3848)
Fixes #3839
2017-03-03 17:07:17 -08:00
Harshavardhana 05e53f1b34 api: CopyObjectPart was copying wrong offsets due to shadowing. (#3838)
startOffset was re-assigned to '0' so it would end up
copying wrong content ignoring the requested startOffset.

This also fixes the corruption issue we observed while
using docker registry.

Fixes https://github.com/docker/distribution/issues/2205

Also fixes #3842 - incorrect routing.
2017-03-03 16:32:04 -08:00
Anis Elleuch 0c8c463a63 tests: Fix web handlers testing with faulty disks (#3845) 2017-03-03 15:46:28 -08:00
Aditya Manthramurthy 6df7bc42b8 Fix check for bucket name: (#3832)
* Do not allow bucket names with adjacent hypen and periods.
* Improve performance by eliminating the usage of regular expressions.
2017-03-03 10:23:41 -08:00
Anis Elleuch 6c00a57a7c quick: Add yaml format support (#3833)
quick Save() and Load() infers config file's format from
file name extension.
2017-03-03 10:22:09 -08:00
Harshavardhana bc52d911ef api: Increase the maximum object size limit from 5GiB to 16GiB. (#3834)
The globalMaxObjectSize limit is instilled in S3 spec perhaps
due to certain limitations on S3 infrastructure. For minio we
don't have such limitations and we can stream a larger file
instead.

So we are going to bump this limit to 16GiB.

Fixes #3825
2017-03-03 10:14:17 -08:00
Anis Elleuch 28c53a3555 obj: Make checkBucketExist() returns all errors (#3843)
This function was returning BucketNotFound for all errors
which at least hides the fact that disks could be corrupted.
This commit fixes the behavior by returning all errors that,
are, by the way, Object API errors.
2017-03-03 10:12:43 -08:00
Harshavardhana e5d4e7aa9d web: Validate if bucket names are reserved (#3841)
Both '.minio.sys' and 'minio' should be never allowed
to be created from web-ui and then fail to list it
by filtering them out.

Fixes #3840
2017-03-03 03:01:42 -08:00
Anis Elleuch cddc684559 admin: Set Config returns errSet and errMsg (#3822)
There is no way to see if a node encountered an error
when trying to set a new config set, this commit adds
a bool errSet field.
2017-03-03 02:53:48 -08:00
Zejun Li 32d0d3d4ac Enhanced newObjectLayerFn (#3837) 2017-03-03 01:07:45 -08:00
Bala FA 98d17d2a97 Remove globalQuiet and globalConfigDir global variables (#3830) 2017-03-02 14:21:30 -08:00
Bala FA 208dd15245 Remove globalMaxCacheSize and globalCacheExpiry variables (#3826)
This patch fixes below

* Remove global variables globalMaxCacheSize and globalCacheExpiry.
* Make global variables into constant in objcache package.
2017-03-02 10:34:37 -08:00
Anis Elleuch a179fc9658 quick: Simplify Load() and CheckVersion() (#3831) 2017-03-02 10:29:06 -08:00
Zejun Li d1afd16955 Using RWMutex to guard closing and listeners (#3829) 2017-03-02 10:00:22 -08:00
Bala FA 2348ae7a19 Make default values as constants (#3828) 2017-03-02 04:58:39 -08:00
Bala FA 480ea826dc Move rlimit functions into sys package. (#3824)
This patch addresses below

* go build works for bsd family
* probe total RAM size for bsd family
* make unit testable functions
2017-03-01 21:51:57 -08:00
Aditya Manthramurthy 09e9fd745c Close client connection after checking for release update (#3820) 2017-03-01 09:18:55 -08:00
Anis Elleuch 77c1998a38 config: Fix creating new config with wrong version (#3821)
Simplify a little config code to avoid making mistake
next time.
2017-03-01 09:17:04 -08:00
Krishna Srinivas 91cf54f895 web-handlers: Support removal of multiple objects at once. (#3810) 2017-02-28 19:07:28 -08:00
Karthic Rao 2b0ed21f08 tests: Fix test server init - cleanup (#3806) 2017-02-28 18:05:52 -08:00
Harshavardhana 472fa4a6ca api: Multi object delete should be protected. (#3814)
Add missing protection from deleting multiple objects
in parallel. Currently we are deleting objects without
proper locking through this API.

This can cause significant amount of races.
2017-02-28 18:00:24 -08:00
Bala FA 097cec676a fix: Set globalMaxCacheSize to allowable value. (#3816)
If memory resource limit and total RAM are more than 8GiB, either 50%
of memory resource limit or total RAM is set to globalMaxCacheSize.
2017-02-28 16:51:52 -08:00
Rushan fcad4a44fd Browser: Remove duplicate object entries while sorting (#3813) 2017-02-28 13:11:34 -08:00
Anis Elleuch 9b3c014bab config: Add browser parameter (#3807)
browser new parameter receives "on" or "off" parameter which is similar
to MINIO_BROWSER
2017-02-27 14:59:53 -08:00
Krishnan Parthasarathi c9619673fb Implement SetConfig admin API handler. (#3792) 2017-02-27 11:40:27 -08:00
Anis Elleuch dce0345f8f Set disk to nil after write which needs quorum (#3795)
Ignore a disk which wasn't able to successfully perform an action to
avoid eventual perturbations when the disk comes back in the middle
of write change.
2017-02-26 11:58:32 -08:00
Anis Elleuch 461b2bbd37 admin: Move SetCredentials from Service to Generic (#3805)
Setting credentials doesn't belong to service management API
anymore.
2017-02-25 11:06:08 -08:00
Bala FA 69777b654e event: use common initialization logic (#3798)
Previously creating and adding targets for each notification type was
repeated.  This patch fixes it.
2017-02-24 18:27:52 -08:00
Harshavardhana 70d2cb5f4d rpc: Remove time check for each RPC calls. (#3804)
This removal comes to avoid some redundant requirements
which are adding more problems on a production setup.

Here are the list of checks for time as they happen

 - Fresh connect (during server startup) - CORRECT
 - A reconnect after network disconnect - CORRECT
 - For each RPC call - INCORRECT.

Verifying time for each RPC aggravates a situation
where a RPC call is rejected in a sequence of events
due to enough load on a production setup. 3 second
might not be enough time window for the call to be
initiated and received by the server.
2017-02-24 18:26:56 -08:00
Harshavardhana cff45db1b9 cli: Use ADDRESS:PORT to clarify --address behavior (#3803)
Currently we document as IP:PORT which doesn't provide
if someone can use HOSTNAME:PORT. This is a change
to clarify this by calling it as ADDRESS:PORT which
encompasses both a HOSTNAME and an IP.

Fixes #3799
2017-02-24 14:19:20 -08:00
Harshavardhana bcc5b6e1ef xl: Rename getOrderedDisks as shuffleDisks appropriately. (#3796)
This PR is for readability cleanup

- getOrderedDisks as shuffleDisks
- getOrderedPartsMetadata as shufflePartsMetadata

Distribution is now a second argument instead being the
primary input argument for brevity.

Also change the usage of type casted int64(0), instead
rely on direct type reference as `var variable int64` everywhere.
2017-02-24 09:20:40 -08:00
Harshavardhana 25b5a0534f browser: Update ui-assets and fix the copyright header. (#3790) 2017-02-22 17:27:26 -08:00
Rushan 52d6678bf0 Browser: Implement multi select user interface for object listings (#3730) 2017-02-22 14:51:38 -08:00
Nitish Tiwari d8950ba7c5 Added server times note and fix Notes rendering for Doctor. (#3787) 2017-02-22 02:12:03 -08:00
Harshavardhana cc28765025 xl/multipart: Make sure to delete temp renamed object. (#3785)
Existing objects before overwrites are renamed to
temp location in completeMultipart. We make sure
that we delete it even if subsequenty calls fail.

Additionally move verifying of parent dir is a
file earlier to fail the entire operation.

Ref #3784
2017-02-21 19:43:44 -08:00
Harshavardhana fe86319c56 ci: For windows builds stick to go1.7.5 (#3786) 2017-02-21 17:24:11 -08:00
Harshavardhana 99a12613a3 update: For source builds look for absolute path. (#3780)
os.Args[0] doesn't point to absolute path we need
use exec.LookPath to find the absolute path before
sending os.Stat().
2017-02-21 01:32:05 -08:00
Nitish Tiwari 097dd7418a Remove unused erasure diagram (#3783) 2017-02-20 21:01:08 -08:00
Nitish Tiwari a7d3ea8c15 Update erasure code image (#3782) 2017-02-20 20:12:21 -08:00
Krishnan Parthasarathi 2745bf2f1f Implement ServerConfig admin REST API (#3741)
Returns a valid config.json of the setup. In case of distributed
setup, it checks if quorum or more number of nodes have the same
config.json.
2017-02-20 12:58:50 -08:00
Anis Elleuch 70d825c608 doc: Small rewrite of bucket events notif intro (#3775) 2017-02-20 12:07:27 -08:00
Harshavardhana 6b68c0170f For streaming signature do not save content-encoding in PutObject() (#3776)
Content-Encoding is set to "aws-chunked" which is an S3 specific
API value which is no meaning for an object. This is how S3
behaves as well for a streaming signature uploaded object.
2017-02-20 12:07:03 -08:00
Aditya Manthramurthy 0a905e1a8a Fix rabbitmq reconnect problem (#3778) 2017-02-20 12:05:21 -08:00
Harshavardhana 9eb8e375c5 cli: Make sure to add --help flag for subcommands. (#3773)
--help is now back and prints properly with command
help template.
2017-02-19 20:46:06 -08:00
Harshavardhana 7ea1de8245 copyObject: Be case sensitive for windows only server. (#3766)
For case sensitive platforms we should honor case.

Fixes #3765

```
1) python s3cmd -c s3cfg_localminio put logo.png s3://testbucket/xyz/etc2/logo.PNG

2) python s3cmd -c s3cfg_localminio ls s3://testbucket/xyz/etc2/
2017-02-18 10:58     22059   s3://testbucket/xyz/etc2/logo.PNG

3) python s3cmd -c s3cfg_localminio cp s3://testbucket/xyz/etc2/logo.PNG s3://testbucket/xyz/etc2/logo.png
remote copy: 's3://testbucket/xyz/etc2/logo.PNG' -> 's3://testbucket/xyz/etc2/logo.png'

4) python s3cmd -c s3cfg_localminio ls s3://testbucket/xyz/etc2/
2017-02-18 10:58     22059   s3://testbucket/xyz/etc2/logo.PNG
2017-02-18 11:10     22059   s3://testbucket/xyz/etc2/logo.png
```
2017-02-18 13:41:59 -08:00
Anis Elleuch 54a18592e9 flags: Fix --version output (#3772) 2017-02-18 13:41:33 -08:00
Anis Elleuch 7e84c7427d server-mux: Rewrite graceful shutdown mechanism (#3771)
Old code uses waitgroup Add() and Wait() in different threads,
which eventually can lead to a race.
2017-02-18 13:28:54 -08:00
Bala FA d12f3e06b1 config-old: Use interface to avoid code repetition. (#3769) 2017-02-18 10:45:37 -08:00
Harshavardhana 0137ff498a auth/rpc: Token can be concurrently edited protect it. (#3764)
Make sure we protect when we access `authToken` in authClient.

Fixes #3761
2017-02-18 03:15:42 -08:00
Harshavardhana 34d9a6b46a Make sure client initializes to proper lock RPC path. (#3763)
Fixes a regression introduced in previous commit.
2017-02-18 02:52:11 -08:00
Harshavardhana 50b4e54a75 fs: Do not return reservedBucket names in ListBuckets() (#3754)
Make sure to skip reserved bucket names in `ListBuckets()`
current code didn't skip this properly and also generalize
this behavior for both XL and FS.
2017-02-16 14:52:14 -08:00
Harshavardhana 8816b08aae Fix the systemd config path to the new URL 2017-02-15 21:28:06 -08:00
Harshavardhana 271e3ecde5 Fix tests from cli changes 2017-02-15 18:05:55 -08:00
1319 changed files with 317547 additions and 16686 deletions
+2 -2
View File
@@ -26,8 +26,8 @@
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used:
* Version used (`minio version`):
* Environment name and version (e.g. nginx 1.9.1):
* Server type and version:
* Operating System and version:
* Operating System and version (`uname -a`):
* Link to your project:
+4
View File
@@ -0,0 +1,4 @@
{
"asi": true,
"esnext": true
}
+20 -2
View File
@@ -1,6 +1,9 @@
go_import_path: github.com/minio/minio
sudo: required
services:
- docker
dist: trusty
language: go
@@ -13,12 +16,27 @@ env:
- ARCH=i686
script:
## Run all the tests
- make
- make test GOFLAGS="-timeout 20m -race -v"
- diff -au <(gofmt -d cmd) <(printf "")
- diff -au <(gofmt -d pkg) <(printf "")
- make test GOFLAGS="-timeout 15m -race -v"
- make coverage
# Refer https://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/
# push image
- >
if [ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && [ "$ARCH" == "x86_64" ]; then
docker run --rm --privileged multiarch/qemu-user-static:register --reset
docker build -t minio/minio:edge-armhf . -f Dockerfile.armhf
docker build -t minio/minio:edge-aarch64 . -f Dockerfile.aarch64
docker login -u="$DOCKER_USER" -p="$DOCKER_PASS"
docker push minio/minio:edge-armhf
docker push minio/minio:edge-aarch64
fi
after_success:
- bash <(curl -s https://codecov.io/bash)
go:
- 1.7.4
- 1.7.5
+24 -11
View File
@@ -1,17 +1,30 @@
FROM golang:1.7-alpine
FROM alpine:3.5
WORKDIR /go/src/app
MAINTAINER Minio Inc <dev@minio.io>
COPY . /go/src/app
ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
RUN \
apk add --no-cache git && \
go-wrapper download && \
go-wrapper install -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
mkdir -p /export/docker && \
rm -rf /go/pkg /go/src && \
apk del git
WORKDIR /go/src/github.com/minio/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps git go musl-dev && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
go get -v -d github.com/minio/minio && \
cd /go/src/github.com/minio/minio && \
go install -v -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
rm -rf /go/pkg /go/src /usr/local/go && apk del .build-deps
EXPOSE 9000
ENTRYPOINT ["minio"]
COPY buildscripts/docker-entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/export"]
CMD ["minio"]
+30
View File
@@ -0,0 +1,30 @@
FROM resin/aarch64-alpine:3.5
MAINTAINER Minio Inc <dev@minio.io>
ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
WORKDIR /go/src/github.com/minio/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps git go musl-dev && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
go get -v -d github.com/minio/minio && \
cd /go/src/github.com/minio/minio && \
go install -v -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
rm -rf /go/pkg /go/src /usr/local/go && apk del .build-deps
EXPOSE 9000
COPY buildscripts/docker-entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/export"]
CMD ["minio"]
+30
View File
@@ -0,0 +1,30 @@
FROM resin/armhf-alpine:3.5
MAINTAINER Minio Inc <dev@minio.io>
ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
WORKDIR /go/src/github.com/minio/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps git go musl-dev && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
go get -v -d github.com/minio/minio && \
cd /go/src/github.com/minio/minio && \
go install -v -ldflags "$(go run buildscripts/gen-ldflags.go)" && \
rm -rf /go/pkg /go/src /usr/local/go && apk del .build-deps
EXPOSE 9000
COPY buildscripts/docker-entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/export"]
CMD ["minio"]
+26
View File
@@ -0,0 +1,26 @@
FROM alpine:3.5
MAINTAINER Minio Inc <dev@minio.io>
COPY buildscripts/docker-entrypoint.sh buildscripts/healthcheck.sh /usr/bin/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps curl && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
curl https://dl.minio.io/server/minio/release/linux-amd64/minio > /usr/bin/minio && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck.sh
EXPOSE 9000
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/export"]
HEALTHCHECK --interval=30s --timeout=5s \
CMD /usr/bin/healthcheck.sh
CMD ["minio"]
+25
View File
@@ -0,0 +1,25 @@
FROM resin/aarch64-alpine:3.5
MAINTAINER Minio Inc <dev@minio.io>
COPY buildscripts/docker-entrypoint.sh buildscripts/healthcheck.sh /usr/bin/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps curl && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
curl https://dl.minio.io/server/minio/release/linux-arm64/minio > /usr/bin/minio && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck.sh
EXPOSE 9000
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/export"]
HEALTHCHECK --interval=30s --timeout=5s \
CMD /usr/bin/healthcheck.sh
CMD ["minio"]
+25
View File
@@ -0,0 +1,25 @@
FROM resin/armhf-alpine:3.5
MAINTAINER Minio Inc <dev@minio.io>
COPY buildscripts/docker-entrypoint.sh buildscripts/healthcheck.sh /usr/bin/
RUN \
apk add --no-cache ca-certificates && \
apk add --no-cache --virtual .build-deps curl && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
curl https://dl.minio.io/server/minio/release/linux-arm/minio > /usr/bin/minio && \
chmod +x /usr/bin/minio && \
chmod +x /usr/bin/docker-entrypoint.sh && \
chmod +x /usr/bin/healthcheck.sh
EXPOSE 9000
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
VOLUME ["/export"]
HEALTHCHECK --interval=30s --timeout=5s \
CMD /usr/bin/healthcheck.sh
CMD ["minio"]
+41 -83
View File
@@ -1,102 +1,50 @@
LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
PWD := $(shell pwd)
GOPATH := $(shell go env GOPATH)
LDFLAGS := $(shell go run buildscripts/gen-ldflags.go)
BUILD_LDFLAGS := '$(LDFLAGS)'
TAG := latest
HOST ?= $(shell uname)
CPU ?= $(shell uname -m)
# if no host is identifed (no uname tool)
# we assume a Linux-64bit build
ifeq ($(HOST),)
HOST = Linux
endif
# identify CPU
ifeq ($(CPU), x86_64)
HOST := $(HOST)64
else
ifeq ($(CPU), amd64)
HOST := $(HOST)64
else
ifeq ($(CPU), i686)
HOST := $(HOST)32
endif
endif
endif
#############################################
# now we find out the target OS for
# which we are going to compile in case
# the caller didn't yet define OS himself
ifndef (OS)
ifeq ($(HOST), Linux64)
arch = gcc
else
ifeq ($(HOST), Linux32)
arch = 32
else
ifeq ($(HOST), Darwin64)
arch = clang
else
ifeq ($(HOST), Darwin32)
arch = clang
else
ifeq ($(HOST), FreeBSD64)
arch = gcc
endif
endif
endif
endif
endif
endif
all: install
all: build
checks:
@echo "Checking deps:"
@echo "Check deps"
@(env bash $(PWD)/buildscripts/checkdeps.sh)
@echo "Checking project is in GOPATH"
@(env bash $(PWD)/buildscripts/checkgopath.sh)
getdeps: checks
@echo "Installing golint:" && go get -u github.com/golang/lint/golint
@echo "Installing gocyclo:" && go get -u github.com/fzipp/gocyclo
@echo "Installing deadcode:" && go get -u github.com/remyoudompheng/go-misc/deadcode
@echo "Installing misspell:" && go get -u github.com/client9/misspell/cmd/misspell
@echo "Installing ineffassign:" && go get -u github.com/gordonklaus/ineffassign
@echo "Installing golint" && go get -u github.com/golang/lint/golint
@echo "Installing gocyclo" && go get -u github.com/fzipp/gocyclo
@echo "Installing deadcode" && go get -u github.com/remyoudompheng/go-misc/deadcode
@echo "Installing misspell" && go get -u github.com/client9/misspell/cmd/misspell
@echo "Installing ineffassign" && go get -u github.com/gordonklaus/ineffassign
verifiers: vet fmt lint cyclo spelling
verifiers: getdeps vet fmt lint cyclo spelling
vet:
@echo "Running $@:"
@go tool vet -all ./cmd
@go tool vet -all ./pkg
@go tool vet -shadow=true ./cmd
@go tool vet -shadow=true ./pkg
@echo "Running $@"
@go tool vet -atomic -bool -copylocks -nilfunc -printf -shadow -rangeloops -unreachable -unsafeptr -unusedresult cmd
@go tool vet -atomic -bool -copylocks -nilfunc -printf -shadow -rangeloops -unreachable -unsafeptr -unusedresult pkg
fmt:
@echo "Running $@:"
@gofmt -s -l cmd
@gofmt -s -l pkg
@echo "Running $@"
@gofmt -d cmd
@gofmt -d pkg
lint:
@echo "Running $@:"
@echo "Running $@"
@${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/cmd...
@${GOPATH}/bin/golint -set_exit_status github.com/minio/minio/pkg...
ineffassign:
@echo "Running $@:"
@echo "Running $@"
@${GOPATH}/bin/ineffassign .
cyclo:
@echo "Running $@:"
@echo "Running $@"
@${GOPATH}/bin/gocyclo -over 100 cmd
@${GOPATH}/bin/gocyclo -over 100 pkg
build: getdeps verifiers $(UI_ASSETS)
deadcode:
@${GOPATH}/bin/deadcode
@@ -105,33 +53,43 @@ spelling:
@${GOPATH}/bin/misspell -error `find pkg/`
@${GOPATH}/bin/misspell -error `find docs/`
test: build
@echo "Running all minio testing:"
# Builds minio, runs the verifiers then runs the tests.
check: test
test: verifiers build
@echo "Running all minio testing"
@go test $(GOFLAGS) .
@go test $(GOFLAGS) github.com/minio/minio/cmd...
@go test $(GOFLAGS) github.com/minio/minio/pkg...
coverage: build
@echo "Running all coverage for minio:"
@echo "Running all coverage for minio"
@./buildscripts/go-coverage.sh
gomake-all: build
@echo "Installing minio:"
@go build --ldflags $(BUILD_LDFLAGS) -o $(GOPATH)/bin/minio
# Builds minio locally.
build:
@echo "Building minio to $(PWD)/minio ..."
@CGO_ENABLED=0 go build --ldflags $(BUILD_LDFLAGS) -o $(PWD)/minio
pkg-add:
${GOPATH}/bin/govendor add $(PKG)
@echo "Adding new package $(PKG)"
@${GOPATH}/bin/govendor add $(PKG)
pkg-update:
${GOPATH}/bin/govendor update $(PKG)
@echo "Updating new package $(PKG)"
@${GOPATH}/bin/govendor update $(PKG)
pkg-remove:
${GOPATH}/bin/govendor remove $(PKG)
@echo "Remove new package $(PKG)"
@${GOPATH}/bin/govendor remove $(PKG)
pkg-list:
@$(GOPATH)/bin/govendor list
install: gomake-all
# Builds minio and installs it to $GOPATH/bin.
install: build
@echo "Installing minio at $(GOPATH)/bin/minio ..."
@cp $(PWD)/minio $(GOPATH)/bin/minio
@echo "Check 'minio -h' for help."
release: verifiers
@MINIO_RELEASE=RELEASE ./buildscripts/build.sh
@@ -140,7 +98,7 @@ experimental: verifiers
@MINIO_RELEASE=EXPERIMENTAL ./buildscripts/build.sh
clean:
@echo "Cleaning up all the generated files:"
@echo "Cleaning up all the generated files"
@find . -name '*.test' | xargs rm -fv
@rm -rf build
@rm -rf release
+28 -15
View File
@@ -18,19 +18,26 @@ docker run -p 9000:9000 minio/minio:edge server /export
```
Please visit Minio Docker quickstart guide for more [here](https://docs.minio.io/docs/minio-docker-quickstart-guide)
## OS X
## macOS
### Homebrew
Install minio packages using [Homebrew](http://brew.sh/)
```sh
brew install minio
brew install minio/stable/minio
minio server ~/Photos
```
#### Note
If you previously installed minio using `brew install minio` then reinstall minio from `minio/stable/minio` official repo. Homebrew builds are unstable due to golang 1.8 bugs.
```
brew uninstall minio
brew install minio/stable/minio
```
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|Apple OS X|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio|
|Apple macOS|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio |
```sh
chmod 755 minio
./minio server ~/Photos
@@ -40,11 +47,11 @@ chmod 755 minio
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|GNU/Linux|64-bit Intel|https://dl.minio.io/server/minio/release/linux-amd64/minio|
||32-bit Intel|https://dl.minio.io/server/minio/release/linux-386/minio|
||32-bit ARM|https://dl.minio.io/server/minio/release/linux-arm/minio|
||64-bit ARM|https://dl.minio.io/server/minio/release/linux-arm64/minio|
||32-bit ARMv6|https://dl.minio.io/server/minio/release/linux-arm6vl/minio|
|GNU/Linux|64-bit Intel|https://dl.minio.io/server/minio/release/linux-amd64/minio |
| |32-bit Intel|https://dl.minio.io/server/minio/release/linux-386/minio |
| |32-bit ARM|https://dl.minio.io/server/minio/release/linux-arm/minio |
| |64-bit ARM|https://dl.minio.io/server/minio/release/linux-arm64/minio |
| |32-bit ARMv6|https://dl.minio.io/server/minio/release/linux-arm6vl/minio |
```sh
chmod +x minio
./minio server ~/Photos
@@ -54,32 +61,38 @@ chmod +x minio
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|Microsoft Windows|64-bit|https://dl.minio.io/server/minio/release/windows-amd64/minio.exe|
||32-bit|https://dl.minio.io/server/minio/release/windows-386/minio.exe|
|Microsoft Windows|64-bit|https://dl.minio.io/server/minio/release/windows-amd64/minio.exe |
| |32-bit|https://dl.minio.io/server/minio/release/windows-386/minio.exe |
```sh
minio.exe server D:\Photos
```
## FreeBSD
### Port
Install minio packages using [pkg](https://github.com/freebsd/pkg)
```sh
pkg install minio
sysrc minio_enable=yes
sysrc minio_disks=/home/user/Photos
service minio start
```
### Binary Download
| Platform| Architecture | URL|
| ----------| -------- | ------|
|FreeBSD|64-bit|https://dl.minio.io/server/minio/release/freebsd-amd64/minio|
|FreeBSD|64-bit|https://dl.minio.io/server/minio/release/freebsd-amd64/minio |
```sh
chmod 755 minio
./minio server ~/Photos
```
You can run Minio on FreeBSD with FreeNAS storage-backend - see [here](https://docs.minio.io/docs/how-to-run-minio-in-freenas) for more details.
## Install from Source
Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow [How to install Golang](https://docs.minio.io/docs/how-to-install-golang).
```sh
go get -u github.com/minio/minio
```
## Test using Minio Browser
+6 -3
View File
@@ -11,11 +11,12 @@ clone_folder: c:\gopath\src\github.com\minio\minio
# Environment variables
environment:
GOROOT: c:\go17
GOPATH: c:\gopath
# scripts that run after cloning repository
install:
- set PATH=%GOPATH%\bin;c:\go\bin;%PATH%
- set PATH=%GOPATH%\bin;c:\go17\bin;%PATH%
- go version
- go env
- python --version
@@ -23,6 +24,8 @@ install:
# To run your custom scripts instead of automatic MSBuild
build_script:
# Compile
# We need to disable firewall - https://github.com/appveyor/ci/issues/1579#issuecomment-309830648
- ps: Disable-NetFirewallRule -DisplayName 'File and Printer Sharing (SMB-Out)'
- appveyor AddCompilationMessage "Starting Compile"
- cd c:\gopath\src\github.com\minio\minio
- go run buildscripts/gen-ldflags.go > temp.txt
@@ -35,9 +38,9 @@ test_script:
# Unit tests
- ps: Add-AppveyorTest "Unit Tests" -Outcome Running
- mkdir build\coverage
- go test -timeout 20m -v -race github.com/minio/minio/cmd...
- go test -v -timeout 17m -race github.com/minio/minio/cmd...
- go test -v -race github.com/minio/minio/pkg...
- go test -timeout 15m -coverprofile=build\coverage\coverage.txt -covermode=atomic github.com/minio/minio/cmd
- go test -v -coverprofile=build\coverage\coverage.txt -covermode=atomic github.com/minio/minio/cmd
- ps: Update-AppveyorTest "Unit Tests" -Outcome Passed
after_test:
+8 -7
View File
@@ -1,8 +1,9 @@
{
"presets": [
"es2015",
"react"
],
"plugins": ["transform-object-rest-spread"]
}
"presets": [
"es2015",
"react"
],
"plugins": [
"transform-object-rest-spread"
]
}
+1 -1
View File
@@ -27,7 +27,7 @@ go get github.com/elazarl/go-bindata-assetfs/...
yarn release
```
This generates ui-assets.go in the current direcotry. Now do `make` in the parent directory to build the minio binary with the newly generated ``ui-assets.go``
This generates ui-assets.go in the current directory. Now do `make` in the parent directory to build the minio binary with the newly generated ``ui-assets.go``
### Run Minio Browser with live reload
+2 -2
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -13,6 +13,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import 'babel-polyfill'
import './less/main.less'
@@ -44,7 +45,6 @@ import Web from './js/web'
window.Web = Web
import storage from 'local-storage-fallback'
const store = applyMiddleware(thunkMiddleware)(createStore)(reducer)
const Browse = connect(state => state)(_Browse)
const Login = connect(state => state)(_Login)
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+95 -23
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -14,13 +14,9 @@
* limitations under the License.
*/
import url from 'url'
import Moment from 'moment'
import browserHistory from 'react-router/lib/browserHistory'
import web from './web'
import * as utils from './utils'
import storage from 'local-storage-fallback'
import { minioBrowserPrefix } from './constants'
export const SET_WEB = 'SET_WEB'
@@ -28,9 +24,10 @@ export const SET_CURRENT_BUCKET = 'SET_CURRENT_BUCKET'
export const SET_CURRENT_PATH = 'SET_CURRENT_PATH'
export const SET_BUCKETS = 'SET_BUCKETS'
export const ADD_BUCKET = 'ADD_BUCKET'
export const ADD_OBJECT = 'ADD_OBJECT'
export const SET_VISIBLE_BUCKETS = 'SET_VISIBLE_BUCKETS'
export const SET_OBJECTS = 'SET_OBJECTS'
export const APPEND_OBJECTS = 'APPEND_OBJECTS'
export const RESET_OBJECTS = 'RESET_OBJECTS'
export const SET_STORAGE_INFO = 'SET_STORAGE_INFO'
export const SET_SERVER_INFO = 'SET_SERVER_INFO'
export const SHOW_MAKEBUCKET_MODAL = 'SHOW_MAKEBUCKET_MODAL'
@@ -57,6 +54,9 @@ export const SET_SHARE_OBJECT = 'SET_SHARE_OBJECT'
export const DELETE_CONFIRMATION = 'DELETE_CONFIRMATION'
export const SET_PREFIX_WRITABLE = 'SET_PREFIX_WRITABLE'
export const REMOVE_OBJECT = 'REMOVE_OBJECT'
export const CHECKED_OBJECTS_ADD = 'CHECKED_OBJECTS_ADD'
export const CHECKED_OBJECTS_REMOVE = 'CHECKED_OBJECTS_REMOVE'
export const CHECKED_OBJECTS_RESET = 'CHECKED_OBJECTS_RESET'
export const showDeleteConfirmation = (object) => {
return {
@@ -78,11 +78,12 @@ export const hideDeleteConfirmation = () => {
}
}
export const showShareObject = url => {
export const showShareObject = (object, url) => {
return {
type: SET_SHARE_OBJECT,
shareObject: {
url: url,
object,
url,
show: true
}
}
@@ -98,15 +99,17 @@ export const hideShareObject = () => {
}
}
export const shareObject = (object, expiry) => (dispatch, getState) => {
export const shareObject = (object, days, hours, minutes) => (dispatch, getState) => {
const {currentBucket, web} = getState()
let host = location.host
let bucket = currentBucket
if (!web.LoggedIn()) {
dispatch(showShareObject(`${host}/${bucket}/${object}`))
dispatch(showShareObject(object, `${host}/${bucket}/${object}`))
return
}
let expiry = days * 24 * 60 * 60 + hours * 60 * 60 + minutes * 60
web.PresignedGet({
host,
bucket,
@@ -114,7 +117,11 @@ export const shareObject = (object, expiry) => (dispatch, getState) => {
expiry
})
.then(obj => {
dispatch(showShareObject(obj.url))
dispatch(showShareObject(object, obj.url))
dispatch(showAlert({
type: 'success',
message: `Object shared. Expires in ${days} days ${hours} hours ${minutes} minutes.`
}))
})
.catch(err => {
dispatch(showAlert({
@@ -235,15 +242,28 @@ export const setVisibleBuckets = visibleBuckets => {
}
}
export const setObjects = (objects, marker, istruncated) => {
const appendObjects = (objects, marker, istruncated) => {
return {
type: SET_OBJECTS,
type: APPEND_OBJECTS,
objects,
marker,
istruncated
}
}
export const setObjects = (objects) => {
return {
type: SET_OBJECTS,
objects,
}
}
export const resetObjects = () => {
return {
type: RESET_OBJECTS
}
}
export const setCurrentBucket = currentBucket => {
return {
type: SET_CURRENT_BUCKET,
@@ -304,14 +324,14 @@ export const listObjects = () => {
marker: marker
})
.then(res => {
let objects = res.objects
let objects = res.objects
if (!objects)
objects = []
objects = objects.map(object => {
object.name = object.name.replace(`${currentPath}`, '');
return object
})
dispatch(setObjects(objects, res.nextmarker, res.istruncated))
object.name = object.name.replace(`${currentPath}`, '');
return object
})
dispatch(appendObjects(objects, res.nextmarker, res.istruncated))
dispatch(setPrefixWritable(res.writable))
dispatch(setLoadBucket(''))
dispatch(setLoadPath(''))
@@ -332,7 +352,7 @@ export const listObjects = () => {
export const selectPrefix = prefix => {
return (dispatch, getState) => {
const {currentBucket, web} = getState()
dispatch(setObjects([], "", false))
dispatch(resetObjects())
dispatch(setLoadPath(prefix))
web.ListObjects({
bucketName: currentBucket,
@@ -344,10 +364,10 @@ export const selectPrefix = prefix => {
if (!objects)
objects = []
objects = objects.map(object => {
object.name = object.name.replace(`${prefix}`, '');
return object
})
dispatch(setObjects(
object.name = object.name.replace(`${prefix}`, '');
return object
})
dispatch(appendObjects(
objects,
res.nextmarker,
res.istruncated
@@ -410,6 +430,37 @@ export const setLoginError = () => {
}
}
export const downloadSelected = (url, req, xhr) => {
return (dispatch) => {
var anchor = document.createElement('a')
document.body.appendChild(anchor);
xhr.open('POST', url, true)
xhr.responseType = 'blob'
xhr.onload = function(e) {
if (this.status == 200) {
dispatch(checkedObjectsReset())
var blob = new Blob([this.response], {
type: 'octet/stream'
})
var blobUrl = window.URL.createObjectURL(blob);
var separator = req.prefix.length > 1 ? '-' : ''
anchor.href = blobUrl
anchor.download = req.bucketName+separator+req.prefix.slice(0, -1)+'.zip';
anchor.click()
window.URL.revokeObjectURL(blobUrl)
anchor.remove()
}
};
xhr.send(JSON.stringify(req));
}
}
export const uploadFile = (file, xhr) => {
return (dispatch, getState) => {
const {currentBucket, currentPath} = getState()
@@ -563,3 +614,24 @@ export const setPolicies = (policies) => {
policies
}
}
export const checkedObjectsAdd = (objectName) => {
return {
type: CHECKED_OBJECTS_ADD,
objectName
}
}
export const checkedObjectsRemove = (objectName) => {
return {
type: CHECKED_OBJECTS_REMOVE,
objectName
}
}
export const checkedObjectsReset = (objectName) => {
return {
type: CHECKED_OBJECTS_RESET,
objectName
}
}
+144 -47
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -27,7 +27,6 @@ import OverlayTrigger from 'react-bootstrap/lib/OverlayTrigger'
import Tooltip from 'react-bootstrap/lib/Tooltip'
import Dropdown from 'react-bootstrap/lib/Dropdown'
import MenuItem from 'react-bootstrap/lib/MenuItem'
import InputGroup from '../components/InputGroup'
import Dropzone from '../components/Dropzone'
import ObjectsList from '../components/ObjectsList'
@@ -69,7 +68,7 @@ export default class Browse extends React.Component {
memory: res.MinioMemory,
platform: res.MinioPlatform,
runtime: res.MinioRuntime,
envVars: res.MinioEnvVars
info: res.MinioGlobalInfo
})
dispatch(actions.setServerInfo(serverInfo))
})
@@ -151,7 +150,16 @@ export default class Browse extends React.Component {
if (prefix === currentPath) return
browserHistory.push(utils.pathJoin(currentBucket, encPrefix))
} else {
window.location = `${window.location.origin}/minio/download/${currentBucket}/${encPrefix}?token=${storage.getItem('token')}`
// Download the selected file.
web.CreateURLToken()
.then(res => {
let url = `${window.location.origin}/minio/download/${currentBucket}/${encPrefix}?token=${res.token}`
window.location = url
})
.catch(err => dispatch(actions.showAlert({
type: 'danger',
message: err.message
})))
}
}
@@ -227,14 +235,29 @@ export default class Browse extends React.Component {
}
removeObject() {
const {web, dispatch, currentPath, currentBucket, deleteConfirmation} = this.props
const {web, dispatch, currentPath, currentBucket, deleteConfirmation, checkedObjects} = this.props
let objects = []
if (checkedObjects.length > 0) {
objects = checkedObjects.map(obj => `${currentPath}${obj}`)
} else {
objects = [deleteConfirmation.object]
}
web.RemoveObject({
bucketName: currentBucket,
objectName: deleteConfirmation.object
bucketname: currentBucket,
objects: objects
})
.then(() => {
this.hideDeleteConfirmation()
dispatch(actions.removeObject(deleteConfirmation.object))
if (checkedObjects.length > 0) {
for (let i = 0; i < checkedObjects.length; i++) {
dispatch(actions.removeObject(checkedObjects[i].replace(currentPath, '')))
}
dispatch(actions.checkedObjectsReset())
} else {
let delObject = deleteConfirmation.object.replace(currentPath, '')
dispatch(actions.removeObject(delObject))
}
})
.catch(e => dispatch(actions.showAlert({
type: 'danger',
@@ -262,7 +285,8 @@ export default class Browse extends React.Component {
shareObject(e, object) {
e.preventDefault()
const {dispatch} = this.props
dispatch(actions.shareObject(object))
// let expiry = 5 * 24 * 60 * 60 // 5 days expiry by default
dispatch(actions.shareObject(object, 5, 0, 0))
}
hideShareObjectModal() {
@@ -354,18 +378,71 @@ export default class Browse extends React.Component {
this.refs.copyTextInput.select()
}
handleExpireValue(targetInput, inc) {
inc === -1 ? this.refs[targetInput].stepDown(1) : this.refs[targetInput].stepUp(1)
handleExpireValue(targetInput, inc, object) {
let value = this.refs[targetInput].value
let maxValue = (targetInput == 'expireHours') ? 23 : (targetInput == 'expireMins') ? 59 : (targetInput == 'expireDays') ? 7 : 0
value = isNaN(value) ? 0 : value
// Use custom step count to support browser Edge
if((inc === -1)) {
if(value != 0) {
value--
}
}
else {
if(value != maxValue) {
value++
}
}
this.refs[targetInput].value = value
// Reset hours and mins when days reaches it's max value
if (this.refs.expireDays.value == 7) {
this.refs.expireHours.value = 0
this.refs.expireMins.value = 0
}
if (this.refs.expireDays.value + this.refs.expireHours.value + this.refs.expireMins.value == 0) {
this.refs.expireDays.value = 7
}
const {dispatch} = this.props
dispatch(actions.shareObject(object, this.refs.expireDays.value, this.refs.expireHours.value, this.refs.expireMins.value))
}
checkObject(e, objectName) {
const {dispatch} = this.props
e.target.checked ? dispatch(actions.checkedObjectsAdd(objectName)) : dispatch(actions.checkedObjectsRemove(objectName))
}
downloadSelected() {
const {dispatch, web} = this.props
let req = {
bucketName: this.props.currentBucket,
objects: this.props.checkedObjects,
prefix: this.props.currentPath
}
web.CreateURLToken()
.then(res => {
let requestUrl = location.origin + "/minio/zip?token=" + res.token
this.xhr = new XMLHttpRequest()
dispatch(actions.downloadSelected(requestUrl, req, this.xhr))
})
.catch(err => dispatch(actions.showAlert({
type: 'danger',
message: err.message
})))
}
clearSelected() {
const {dispatch} = this.props
dispatch(actions.checkedObjectsReset())
}
render() {
const {total, free} = this.props.storageInfo
const {showMakeBucketModal, alert, sortNameOrder, sortSizeOrder, sortDateOrder, showAbout, showBucketPolicy} = this.props
const {showMakeBucketModal, alert, sortNameOrder, sortSizeOrder, sortDateOrder, showAbout, showBucketPolicy, checkedObjects} = this.props
const {version, memory, platform, runtime} = this.props.serverInfo
const {sidebarStatus} = this.props
const {showSettings} = this.props
@@ -420,21 +497,24 @@ export default class Browse extends React.Component {
}
if (web.LoggedIn()) {
storageUsageDetails = <div className="feh-usage">
<div className="fehu-chart">
<div style={ { width: usedPercent } }></div>
</div>
<ul>
<li>
Used:
{ humanize.filesize(total - free) }
</li>
<li className="pull-right">
Free:
{ humanize.filesize(total - used) }
</li>
</ul>
</div>
if (!(used === 0 && free === 0)) {
storageUsageDetails = <div className="feh-usage">
<div className="fehu-chart">
<div style={ { width: usedPercent } }></div>
</div>
<ul>
<li>
<span>Used: </span>
{ humanize.filesize(total - free) }
</li>
<li className="pull-right">
<span>Free: </span>
{ humanize.filesize(total - used) }
</li>
</ul>
</div>
}
}
@@ -490,6 +570,14 @@ export default class Browse extends React.Component {
clickOutside={ this.hideSidebar.bind(this) }
showPolicy={ this.showBucketPolicy.bind(this) } />
<div className="fe-body">
<div className={ 'list-actions' + (classNames({
' list-actions-toggled': checkedObjects.length > 0
})) }>
<span className="la-label"><i className="fa fa-check-circle" /> { checkedObjects.length } Objects selected</span>
<span className="la-actions pull-right"><button onClick={ this.downloadSelected.bind(this) }> Download all as zip </button></span>
<span className="la-actions pull-right"><button onClick={ this.showDeleteConfirmation.bind(this) }> Delete selected </button></span>
<i className="la-close fa fa-times" onClick={ this.clearSelected.bind(this) }></i>
</div>
<Dropzone>
{ alertBox }
<header className="fe-header-mobile hidden-lg hidden-md">
@@ -515,7 +603,8 @@ export default class Browse extends React.Component {
</header>
<div className="feb-container">
<header className="fesl-row" data-type="folder">
<div className="fesl-item fi-name" onClick={ this.sortObjectsByName.bind(this) } data-sort="name">
<div className="fesl-item fesl-item-icon"></div>
<div className="fesl-item fesl-item-name" onClick={ this.sortObjectsByName.bind(this) } data-sort="name">
Name
<i className={ classNames({
'fesli-sort': true,
@@ -524,7 +613,7 @@ export default class Browse extends React.Component {
'fa-sort-alpha-asc': !sortNameOrder
}) } />
</div>
<div className="fesl-item fi-size" onClick={ this.sortObjectsBySize.bind(this) } data-sort="size">
<div className="fesl-item fesl-item-size" onClick={ this.sortObjectsBySize.bind(this) } data-sort="size">
Size
<i className={ classNames({
'fesli-sort': true,
@@ -533,7 +622,7 @@ export default class Browse extends React.Component {
'fa-sort-amount-asc': !sortSizeOrder
}) } />
</div>
<div className="fesl-item fi-modified" onClick={ this.sortObjectsByDate.bind(this) } data-sort="last-modified">
<div className="fesl-item fesl-item-modified" onClick={ this.sortObjectsByDate.bind(this) } data-sort="last-modified">
Last Modified
<i className={ classNames({
'fesli-sort': true,
@@ -542,7 +631,7 @@ export default class Browse extends React.Component {
'fa-sort-numeric-asc': !sortDateOrder
}) } />
</div>
<div className="fesl-item fi-actions"></div>
<div className="fesl-item fesl-item-actions"></div>
</header>
</div>
<div className="feb-container">
@@ -553,9 +642,11 @@ export default class Browse extends React.Component {
<ObjectsList dataType={ this.dataType.bind(this) }
selectPrefix={ this.selectPrefix.bind(this) }
showDeleteConfirmation={ this.showDeleteConfirmation.bind(this) }
shareObject={ this.shareObject.bind(this) } />
shareObject={ this.shareObject.bind(this) }
checkObject={ this.checkObject.bind(this) }
checkedObjectsArray={ checkedObjects } />
</InfiniteScroll>
<div className="text-center" style={ { display: istruncated ? 'block' : 'none' } }>
<div className="text-center" style={ { display: (istruncated && currentBucket) ? 'block' : 'none' } }>
<span>Loading...</span>
</div>
</div>
@@ -669,11 +760,11 @@ export default class Browse extends React.Component {
</div>
<div className="input-group" style={ { display: web.LoggedIn() ? 'block' : 'none' } }>
<label>
Expires in
Expires in (Max 7 days)
</label>
<div className="set-expire">
<div className="set-expire-item">
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireDays', 1) }></i>
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireDays', 1, shareObject.object) } />
<div className="set-expire-title">
Days
</div>
@@ -682,12 +773,14 @@ export default class Browse extends React.Component {
type="number"
min={ 0 }
max={ 7 }
defaultValue={ 0 } />
defaultValue={ 5 }
readOnly="readOnly"
/>
</div>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireDays', -1) }></i>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireDays', -1, shareObject.object) } />
</div>
<div className="set-expire-item">
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireHours', 1) }></i>
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireHours', 1, shareObject.object) } />
<div className="set-expire-title">
Hours
</div>
@@ -695,30 +788,34 @@ export default class Browse extends React.Component {
<input ref="expireHours"
type="number"
min={ 0 }
max={ 24 }
defaultValue={ 0 } />
max={ 23 }
defaultValue={ 0 }
readOnly="readOnly"
/>
</div>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireHours', -1) }></i>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireHours', -1, shareObject.object) } />
</div>
<div className="set-expire-item">
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireMins', 1) }></i>
<i className="set-expire-increase" onClick={ this.handleExpireValue.bind(this, 'expireMins', 1, shareObject.object) } />
<div className="set-expire-title">
Minutes
</div>
<div className="set-expire-value">
<input ref="expireMins"
type="number"
min={ 1 }
max={ 60 }
defaultValue={ 45 } />
min={ 0 }
max={ 59 }
defaultValue={ 0 }
readOnly="readOnly"
/>
</div>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireMins', -1) }></i>
<i className="set-expire-decrease" onClick={ this.handleExpireValue.bind(this, 'expireMins', -1, shareObject.object) } />
</div>
</div>
</div>
</ModalBody>
<div className="modal-footer">
<CopyToClipboard text={ shareObject.url } onCopy={ this.showMessage.bind(this) }>
<CopyToClipboard text={ window.location.protocol + '//' + shareObject.url } onCopy={ this.showMessage.bind(this) }>
<button className="btn btn-success">
Copy Link
</button>
+2 -2
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016, 2017 Minio, Inc.
* Minio Cloud Storage (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -27,7 +27,7 @@ let BrowserDropdown = ({fullScreenFunc, aboutFunc, settingsFunc, logoutFunc}) =>
</Dropdown.Toggle>
<Dropdown.Menu className="dropdown-menu-right">
<li>
<a target="_blank" href="https://github.com/minio/miniobrowser">Github <i className="fa fa-github"></i></a>
<a target="_blank" href="https://github.com/minio/minio">Github <i className="fa fa-github"></i></a>
</li>
<li>
<a href="" onClick={ fullScreenFunc }>Fullscreen <i className="fa fa-expand"></i></a>
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+3 -2
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -39,11 +39,12 @@ export default class Dropzone extends React.Component {
// won't handle child elements correctly.
const style = {
height: '100%',
borderWidth: '2px',
borderWidth: '0',
borderStyle: 'dashed',
borderColor: '#fff'
}
const activeStyle = {
borderWidth: '2px',
borderColor: '#777'
}
const rejectStyle = {
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -6
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -89,10 +89,6 @@ export default class Login extends React.Component {
{ alertBox }
<div className="l-wrap">
<form onSubmit={ this.handleSubmit.bind(this) }>
<input name="fixBrowser"
autoComplete="username"
type="text"
style={ { display: 'none' } } />
<InputGroup className="ig-dark"
label="Access Key"
id="accessKey"
@@ -102,7 +98,6 @@ export default class Login extends React.Component {
required="required"
autoComplete="username">
</InputGroup>
<input type="text" autoComplete="new-password" style={ { display: 'none' } } />
<InputGroup className="ig-dark"
label="Secret Key"
id="secretKey"
+38 -17
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -20,8 +20,7 @@ import humanize from 'humanize'
import connect from 'react-redux/lib/components/connect'
import Dropdown from 'react-bootstrap/lib/Dropdown'
let ObjectsList = ({objects, currentPath, selectPrefix, dataType, showDeleteConfirmation, shareObject, loadPath}) => {
let ObjectsList = ({objects, currentPath, selectPrefix, dataType, showDeleteConfirmation, shareObject, loadPath, checkObject, checkedObjectsArray}) => {
const list = objects.map((object, i) => {
let size = object.name.endsWith('/') ? '-' : humanize.filesize(object.size)
let lastModified = object.name.endsWith('/') ? '-' : Moment(object.lastModified).format('lll')
@@ -30,29 +29,51 @@ let ObjectsList = ({objects, currentPath, selectPrefix, dataType, showDeleteConf
let deleteButton = ''
if (web.LoggedIn())
deleteButton = <a href="" className="fiad-action" onClick={ (e) => showDeleteConfirmation(e, `${currentPath}${object.name}`) }><i className="fa fa-trash"></i></a>
if (!object.name.endsWith('/')) {
actionButtons = <Dropdown id="fia-dropdown">
<Dropdown.Toggle noCaret className="fia-toggle"></Dropdown.Toggle>
<Dropdown.Menu>
<a href="" className="fiad-action" onClick={ (e) => shareObject(e, `${currentPath}${object.name}`) }><i className="fa fa-copy"></i></a>
{ deleteButton }
</Dropdown.Menu>
</Dropdown>
if (!checkedObjectsArray.length > 0) {
if (!object.name.endsWith('/')) {
actionButtons = <Dropdown id={ "fia-dropdown-" + object.name.replace('.', '-') }>
<Dropdown.Toggle noCaret className="fia-toggle"></Dropdown.Toggle>
<Dropdown.Menu>
<a href="" className="fiad-action" onClick={ (e) => shareObject(e, `${currentPath}${object.name}`) }><i className="fa fa-copy"></i></a>
{ deleteButton }
</Dropdown.Menu>
</Dropdown>
}
}
let activeClass = ''
let isChecked = ''
if (checkedObjectsArray.indexOf(object.name) > -1) {
activeClass = ' fesl-row-selected'
isChecked = true
}
return (
<div key={ i } className={ "fesl-row " + loadingClass } data-type={ dataType(object.name, object.contentType) }>
<div className="fesl-item fi-name">
<div key={ i } className={ "fesl-row " + loadingClass + activeClass } data-type={ dataType(object.name, object.contentType) }>
<div className="fesl-item fesl-item-icon">
<div className="fi-select">
<input type="checkbox"
name={ object.name }
checked={ isChecked }
onChange={ (e) => checkObject(e, object.name) } />
<i className="fis-icon"></i>
<i className="fis-helper"></i>
</div>
</div>
<div className="fesl-item fesl-item-name">
<a href="" onClick={ (e) => selectPrefix(e, `${currentPath}${object.name}`) }>
{ object.name }
</a>
</div>
<div className="fesl-item fi-size">
<div className="fesl-item fesl-item-size">
{ size }
</div>
<div className="fesl-item fi-modified">
<div className="fesl-item fesl-item-modified">
{ lastModified }
</div>
<div className="fesl-item fi-actions">
<div className="fesl-item fesl-item-actions">
{ actionButtons }
</div>
</div>
@@ -72,4 +93,4 @@ export default connect(state => {
currentPath: state.currentPath,
loadPath: state.loadPath
}
})(ObjectsList)
})(ObjectsList)
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -77,4 +77,4 @@ class Policy extends Component {
}
}
export default connect(state => state)(Policy)
export default connect(state => state)(Policy)
+19 -4
View File
@@ -7,6 +7,7 @@ import * as actions from '../actions'
class PolicyInput extends Component {
componentDidMount() {
const {web, dispatch} = this.props
this.prefix.focus()
web.ListAllBucketPolicies({
bucketName: this.props.currentBucket
}).then(res => {
@@ -27,8 +28,23 @@ class PolicyInput extends Component {
handlePolicySubmit(e) {
e.preventDefault()
const {web, dispatch} = this.props
const {web, dispatch, currentBucket} = this.props
let prefix = currentBucket + '/' + this.prefix.value
let policy = this.policy.value
if (!prefix.endsWith('*')) prefix = prefix + '*'
let prefixAlreadyExists = this.props.policies.some(elem => prefix === elem.prefix)
if (prefixAlreadyExists) {
dispatch(actions.showAlert({
type: 'danger',
message: "Policy for this prefix already exists."
}))
return
}
web.SetBucketPolicy({
bucketName: this.props.currentBucket,
prefix: this.prefix.value,
@@ -36,8 +52,7 @@ class PolicyInput extends Component {
})
.then(() => {
dispatch(actions.setPolicies([{
policy: this.policy.value,
prefix: this.prefix.value + '*',
policy, prefix
}, ...this.props.policies]))
this.prefix.value = ''
})
@@ -80,4 +95,4 @@ class PolicyInput extends Component {
}
}
export default connect(state => state)(PolicyInput)
export default connect(state => state)(PolicyInput)
+6 -17
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -34,23 +34,12 @@ class SettingsModal extends React.Component {
let accessKeyEnv = ''
let secretKeyEnv = ''
// Check environment variables first. They may or may not have been
// loaded already; they load in Browse#componentDidMount.
if (serverInfo.envVars) {
serverInfo.envVars.forEach(envVar => {
let keyVal = envVar.split('=')
if (keyVal[0] == 'MINIO_ACCESS_KEY') {
accessKeyEnv = keyVal[1]
} else if (keyVal[0] == 'MINIO_SECRET_KEY') {
secretKeyEnv = keyVal[1]
}
})
}
if (accessKeyEnv != '' || secretKeyEnv != '') {
// Check environment variables first.
if (serverInfo.info.isEnvCreds) {
dispatch(actions.setSettings({
accessKey: accessKeyEnv,
secretKey: secretKeyEnv,
keysReadOnly: true
accessKey: 'xxxxxxxxx',
secretKey: 'xxxxxxxxx',
keysReadOnly: true
}))
} else {
web.GetAuth()
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+28 -12
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -54,9 +54,10 @@ export default (state = {
shareObject: {
show: false,
url: '',
expiry: 604800
object: ''
},
prefixWritable: false
prefixWritable: false,
checkedObjects: []
}, action) => {
let newState = Object.assign({}, state)
switch (action.type) {
@@ -76,16 +77,18 @@ export default (state = {
case actions.SET_CURRENT_BUCKET:
newState.currentBucket = action.currentBucket
break
case actions.APPEND_OBJECTS:
newState.objects = [...newState.objects, ...action.objects]
newState.marker = action.marker
newState.istruncated = action.istruncated
break
case actions.SET_OBJECTS:
if (!action.objects.length) {
newState.objects = []
newState.marker = ""
newState.istruncated = action.istruncated
} else {
newState.objects = [...newState.objects, ...action.objects]
newState.marker = action.marker
newState.istruncated = action.istruncated
}
newState.objects = [...action.objects]
break
case actions.RESET_OBJECTS:
newState.objects = []
newState.marker = ""
newState.istruncated = false
break
case actions.SET_CURRENT_PATH:
newState.currentPath = action.currentPath
@@ -185,6 +188,19 @@ export default (state = {
if (idx == -1) break
newState.objects = [...newState.objects.slice(0, idx), ...newState.objects.slice(idx + 1)]
break
case actions.CHECKED_OBJECTS_ADD:
newState.checkedObjects = [...newState.checkedObjects, action.objectName]
break
case actions.CHECKED_OBJECTS_REMOVE:
let index = newState.checkedObjects.indexOf(action.objectName)
if (index == -1) break
newState.checkedObjects = [...newState.checkedObjects.slice(0, index), ...newState.checkedObjects.slice(index + 1)]
break
case actions.CHECKED_OBJECTS_RESET:
newState.checkedObjects = []
break
}
return newState
}
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
+4 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -112,6 +112,9 @@ export default class Web {
return res
})
}
CreateURLToken() {
return this.makeCall('CreateURLToken')
}
GetBucketPolicy(args) {
return this.makeCall('GetBucketPolicy', args)
}
+4
View File
@@ -35,6 +35,10 @@
width: 100%;
}
.btn-white {
.btn-variant(#fff, darken(@text-color, 20%));
}
.btn-link {
.btn-variant(#eee, #545454);
}
+16 -3
View File
@@ -81,7 +81,7 @@ select.form-control {
width: 100%;
height: 40px;
border: 0;
background: transparent;
background: transparent !important;
text-align: center;
position: relative;
z-index: 1;
@@ -114,8 +114,8 @@ select.form-control {
.ig-dark {
.ig-text {
color: @white;
border-color: rgba(255,255,255,0.1);
color: @white !important;
border-color: rgba(255,255,255,0.1) !important;
}
.ig-helpers {
@@ -183,6 +183,17 @@ select.form-control {
.set-expire {
border: 1px solid @input-border;
margin: 35px 0 30px;
position: relative;
&:before {
content: '';
position: absolute;
left: 0;
top: 0;
width: 100%;
height: 100%;
z-index: 1;
}
}
.set-expire-item {
@@ -191,6 +202,7 @@ select.form-control {
display: table-cell;
width: 1%;
text-align: center;
.user-select(none);
&:not(:last-child) {
border-right: 1px solid @input-border;
@@ -209,6 +221,7 @@ select.form-control {
left: -8px;
input {
.user-select(none);
font-size: 20px;
text-align: center;
position: relative;
+3 -5
View File
@@ -2,14 +2,13 @@
Header
----------------------------*/
.fe-header {
padding: 45px 55px 20px;
@media(min-width: @screen-md-min) {
@media(min-width: (@screen-sm-min - 100)) {
position: relative;
padding: 40px 40px 20px 45px;
}
@media(max-width: (@screen-xs-max - 100)) {
padding: 25px 25px 20px;
padding: 20px;
}
h2 {
@@ -239,4 +238,3 @@
}
+236 -102
View File
@@ -2,17 +2,19 @@
Row
----------------------------*/
.fesl-row {
padding-right: 40px;
padding-top: 5px;
padding-bottom: 5px;
position: relative;
@media (min-width: (@screen-sm-min - 100px)) {
padding: 5px 20px 5px 40px;
display: flex;
flex-flow: row nowrap;
justify-content: space-between;
}
@media(max-width: (@screen-xs-max - 100px)) {
padding: 5px 20px;
}
.clearfix();
}
@@ -20,7 +22,7 @@ header.fesl-row {
@media (min-width:(@screen-sm-min - 100px)) {
margin-bottom: 20px;
border-bottom: 1px solid lighten(@text-muted-color, 20%);
padding-left: 40px;
padding-left: 30px;
.fesl-item,
.fesli-sort {
@@ -42,7 +44,7 @@ header.fesl-row {
font-size: 14px;
}
&:hover:not(.fi-actions) {
&:hover:not(.fesl-item-actions) {
background: lighten(@text-muted-color, 22%);
color: @dark-gray;
@@ -58,54 +60,42 @@ header.fesl-row {
}
}
.list-type(@background, @icon) {
.fis-icon {
background-color: @background;
&:before {
content: @icon;
}
}
}
div.fesl-row {
padding-left: 85px;
border-bottom: 1px solid transparent;
cursor: default;
.transition(background-color);
.transition-duration(500ms);
@media (max-width: (@screen-xs-max - 100px)) {
padding-left: 70px;
padding-right: 45px;
padding: 5px 20px;
}
&:nth-child(even) {
background-color: #fafafa;
}
&:hover {
background-color: #fbf7dc;
}
&[data-type]:before {
font-family: @font-family-icon;
width: 35px;
height: 35px;
text-align: center;
line-height: 35px;
position: absolute;
border-radius: 50%;
font-size: 16px;
left: 50px;
top: 9px;
color: @white;
@media (max-width: (@screen-xs-max - 100px)) {
left: 20px;
&:not(.fesl-row-selected) {
&:nth-child(even) {
background-color: @list-row-even-bg;
}
}
&[data-type="folder"] {
@media (max-width: (@screen-xs-max - 100px)) {
.fesl-item {
&.fi-name {
padding-top: 10px;
padding-bottom: 7px;
}
&:hover {
.fis-icon {
&:before {
.opacity(0)
}
}
&.fi-size,
&.fi-modified {
display: none;
}
.fis-helper {
&:before {
.opacity(1);
}
}
}
@@ -113,54 +103,18 @@ div.fesl-row {
/*--------------------------
Icons
----------------------------*/
&[data-type=folder]:before {
content: '\f114';
background-color: #a1d6dd;
}
&[data-type=pdf]:before {
content: "\f1c1";
background-color: #fa7775;
}
&[data-type=zip]:before {
content: "\f1c6";
background-color: #427089;
}
&[data-type=audio]:before {
content: "\f1c7";
background-color: #009688
}
&[data-type=code]:before {
content: "\f1c9";
background-color: #997867;
}
&[data-type=excel]:before {
content: "\f1c3";
background-color: #64c866;
}
&[data-type=image]:before {
content: "\f1c5";
background-color: #f06292;
}
&[data-type=video]:before {
content: "\f1c8";
background-color: #f8c363;
}
&[data-type=other]:before {
content: "\f016";
background-color: #afafaf;
}
&[data-type=text]:before {
content: "\f0f6";
background-color: #8a8a8a;
}
&[data-type=doc]:before {
content: "\f1c2";
background-color: #2196f5;
}
&[data-type=presentation]:before {
content: "\f1c4";
background-color: #896ea6;
}
&[data-type=folder] { .list-type(#a1d6dd, '\f114'); }
&[data-type=pdf] {.list-type(#fa7775, '\f1c1'); }
&[data-type=zip] { .list-type(#427089, '\f1c6'); }
&[data-type=audio] { .list-type(#009688, '\f1c7'); }
&[data-type=code] { .list-type(#997867, "\f1c9"); }
&[data-type=excel] { .list-type(#f1c3, '\f1c3'); }
&[data-type=image] { .list-type(#f06292, '\f1c5'); }
&[data-type=video] { .list-type(#f8c363, '\f1c8'); }
&[data-type=other] { .list-type(#afafaf, '\f016'); }
&[data-type=text] { .list-type(#8a8a8a, '\f0f6'); }
&[data-type=doc] { .list-type(#2196f5, '\f1c2'); }
&[data-type=presentation] { .list-type(#896ea6, '\f1c4'); }
&.fesl-loading{
&:before {
@@ -180,6 +134,113 @@ div.fesl-row {
}
}
.fesl-row-selected {
background-color: @list-row-selected-bg;
&, .fesl-item a {
color: darken(@text-color, 10%);
}
}
.fi-select {
float: left;
position: relative;
width: 35px;
height: 35px;
margin: 3px 0;
@media(max-width: (@screen-xs-max - 100px)) {
margin-right: 15px;
}
input {
position: absolute;
left: 0;
top: 0;
width: 35px;
height: 35px;
z-index: 20;
opacity: 0;
cursor: pointer;
&:checked {
& ~ .fis-icon {
background-color: #32393F;
&:before {
opacity: 0;
}
}
& ~ .fis-helper {
&:before {
.scale(0);
}
&:after {
.scale(1);
}
}
}
}
}
.fis-icon {
display: inline-block;
vertical-align: top;
border-radius: 50%;
width: 35px;
height: 35px;
.transition(background-color);
.transition-duration(250ms);
&:before {
width: 100%;
height: 100%;
text-align: center;
position: absolute;
border-radius: 50%;
font-family: @font-family-icon;
line-height: 35px;
font-size: 16px;
color: @white;
.transition(all);
.transition-duration(300ms);
font-style: normal;
}
}
.fis-helper {
&:before,
&:after {
position: absolute;
.transition(all);
.transition-duration(250ms);
}
&:before {
content: '';
width: 15px;
height: 15px;
border: 2px solid @white;
z-index: 10;
border-radius: 2px;
top: 10px;
left: 10px;
opacity: 0;
}
&:after {
font-family: @font-family-icon;
content: '\f00c';
top: 8px;
left: 9px;
color: @white;
font-size: 14px;
.scale(0);
}
}
/*--------------------------
Files and Folders
@@ -192,26 +253,26 @@ div.fesl-row {
}
@media(min-width: (@screen-sm-min - 100px)) {
&:not(.fi-actions) {
&:not(.fesl-item-actions):not(.fesl-item-icon) {
text-overflow: ellipsis;
padding: 10px 15px;
white-space: nowrap;
overflow: hidden;
}
&.fi-name {
&.fesl-item-name {
flex: 3;
}
&.fi-size {
&.fesl-item-size {
width: 140px;
}
&.fi-modified {
&.fesl-item-modified {
width: 190px;
}
&.fi-actions {
&.fesl-item-actions {
width: 40px;
}
}
@@ -219,29 +280,29 @@ div.fesl-row {
@media(max-width: (@screen-xs-max - 100px)) {
padding: 0;
&.fi-name {
&.fesl-item-name {
width: 100%;
margin-bottom: 3px;
}
&.fi-size,
&.fi-modified {
&.fesl-item-size,
&.fesl-item-modified {
font-size: 12px;
color: #B5B5B5;
float: left;
}
&.fi-modified {
&.fesl-item-modified {
max-width: 72px;
white-space: nowrap;
overflow: hidden;
}
&.fi-size {
&.fesl-item-size {
margin-right: 10px;
}
&.fi-actions {
&.fesl-item-actions {
position: absolute;
top: 5px;
right: 10px;
@@ -266,7 +327,7 @@ div.fesl-row {
}
}
.fi-actions {
.fesl-item-actions {
.dropdown-menu {
background-color: transparent;
box-shadow: none;
@@ -324,6 +385,79 @@ div.fesl-row {
}
}
.list-actions {
position: fixed;
.translate3d(0, -100%, 0);
.opacity(0);
.transition(all);
.transition-duration(200ms);
padding: 20px 70px 20px 25px;
top: 0;
left: 0;
width: 100%;
background-color: @brand-primary;
z-index: 20;
box-shadow: 0 0 10px rgba(0,0,0,0.3);
text-align: center;
&.list-actions-toggled {
.translate3d(0, 0, 0);
.opacity(1);
}
}
.la-close {
position: absolute;
right: 20px;
top: 0;
color: #fff;
width: 30px;
height: 30px;
border-radius: 50%;
text-align: center;
line-height: 30px !important;
background: rgba(255, 255, 255, 0.1);
font-weight: normal;
bottom: 0;
margin: auto;
cursor: pointer;
&:hover {
background-color: rgba(255, 255, 255, 0.2);
}
}
.la-label {
color: @white;
float: left;
padding: 4px 0;
.fa {
font-size: 22px;
vertical-align: top;
margin-right: 10px;
margin-top: -1px;
}
}
.la-actions {
button {
background-color: transparent;
border: 2px solid rgba(255,255,255,0.9);
color: @white;
border-radius: 2px;
padding: 5px 10px;
font-size: 13px;
.transition(all);
.transition-duration(300ms);
margin-left: 10px;
&:hover {
background-color: @white;
color: @brand-primary;
}
}
}
@-webkit-keyframes fiad-action-anim {
from {
+14
View File
@@ -99,4 +99,18 @@
content: '7 days';
right: 0;
}
}
.modal-aheader {
height: 100px;
&:before {
height: 0 !important;
}
.modal-dialog {
margin: 0;
vertical-align: top;
}
}
+1 -1
View File
@@ -49,4 +49,4 @@
z-index: 1;
-webkit-animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
}
}
+8 -8
View File
@@ -7,7 +7,7 @@
position: fixed;
height: 100%;
overflow: hidden;
padding: 35px;
padding: 25px;
@media(min-width: @screen-md-min) {
.translate3d(0, 0, 0);
@@ -63,15 +63,15 @@
height: ~"calc(100vh - 260px)";
overflow: auto;
padding: 0;
margin: 0 -35px;
margin: 0 -25px;
& li {
position: relative;
& > a {
display: block;
padding: 10px 40px 12px 65px;
.text-overflow();
padding: 10px 45px 12px 55px;
word-wrap: break-word;
&:before {
font-family: FontAwesome;
@@ -79,7 +79,7 @@
font-size: 17px;
position: absolute;
top: 10px;
left: 35px;
left: 25px;
.opacity(0.8);
}
@@ -95,7 +95,7 @@
}
&.active {
background-color: rgba(0, 0, 0, 0.2);
background-color: #282e32;
& > a {
color: @white;
@@ -139,10 +139,10 @@
position: absolute;
top: 0;
right: 0;
width: 40px;
width: 35px;
height: 100%;
cursor: pointer;
background: url(../../img/more-h-light.svg) no-repeat left;
background: url(../../img/more-h-light.svg) no-repeat top 20px left;
}
/* Scrollbar */
+10 -3
View File
@@ -13,7 +13,7 @@
/*--------------------------
File Explorer
----------------------------*/
@fe-sidebar-width : 300px;
@fe-sidebar-width : 320px;
@text-muted-color : #BDBDBD;
@text-strong-color : #333;
@@ -81,7 +81,7 @@
/*-------------------------
Colors
--------------------------*/
@brand-primary: #2196F3;
@brand-primary: #2298f7;
@brand-success: #4CAF50;
@brand-info: #00BCD4;
@brand-warning: #FF9800;
@@ -91,4 +91,11 @@
/*-------------------------
Form
--------------------------*/
@input-border: #eee;
@input-border: #eee;
/*-------------------------
List
--------------------------*/
@list-row-selected-bg: #fbf2bf;
@list-row-even-bg: #fafafa;
+3 -3
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -70,9 +70,9 @@ async.waterfall([
commitId = stdout.replace('\n', '')
if (commitId.length !== 40) throw new Error('commitId invalid : ' + commitId)
assetsFileName = 'ui-assets.go';
var cmd = 'go-bindata-assetfs -pkg miniobrowser -nocompress=true production/...'
var cmd = 'go-bindata-assetfs -pkg browser -nocompress=true production/...'
if (!isProduction) {
cmd = 'go-bindata-assetfs -pkg miniobrowser -nocompress=true dev/...'
cmd = 'go-bindata-assetfs -pkg browser -nocompress=true dev/...'
}
console.log('Running', cmd)
exec(cmd, cb)
+5 -4
View File
@@ -1,5 +1,5 @@
{
"name": "minio-browser",
"name": "browser",
"version": "0.0.1",
"description": "Minio Browser",
"scripts": {
@@ -11,14 +11,14 @@
},
"repository": {
"type": "git",
"url": "https://github.com/minio/miniobrowser"
"url": "https://github.com/minio/minio"
},
"author": "Minio Inc",
"license": "Apache-2.0",
"bugs": {
"url": "https://github.com/minio/miniobrowser/issues"
"url": "https://github.com/minio/minio/issues"
},
"homepage": "https://github.com/minio/miniobrowser",
"homepage": "https://github.com/minio/minio",
"devDependencies": {
"async": "^1.5.2",
"babel-cli": "^6.14.0",
@@ -26,6 +26,7 @@
"babel-loader": "^6.2.5",
"babel-plugin-syntax-object-rest-spread": "^6.13.0",
"babel-plugin-transform-object-rest-spread": "^6.8.0",
"babel-polyfill": "^6.23.0",
"babel-preset-es2015": "^6.14.0",
"babel-preset-react": "^6.11.1",
"babel-register": "^6.14.0",
+65 -55
View File
File diff suppressed because one or more lines are too long
+5 -1
View File
@@ -1,5 +1,5 @@
/*
* Minio Browser (C) 2016 Minio, Inc.
* Minio Cloud Storage (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -71,6 +71,10 @@ var exports = {
target: 'http://localhost:9000',
secure: false
},
'/minio/zip': {
target: 'http://localhost:9000',
secure: false
}
}
},
plugins: [
+1356 -1031
View File
File diff suppressed because it is too large Load Diff
+38 -50
View File
@@ -5,7 +5,7 @@ _init() {
LDFLAGS=$(go run buildscripts/gen-ldflags.go)
# Extract release tag
release_tag=$(echo $LDFLAGS | awk {'print $4'} | cut -f2 -d=)
release_tag=$(echo $LDFLAGS | awk {'print $6'} | cut -f2 -d=)
# Verify release tag.
if [ -z "$release_tag" ]; then
@@ -28,6 +28,7 @@ _init() {
## System binaries
CP=`which cp`
SHASUM=`which shasum`
SHA256SUM="${SHASUM} -a 256"
SED=`which sed`
}
@@ -43,58 +44,36 @@ go_build() {
release_bin="$release_str/$os-$arch/$(basename $package).$release_tag"
# Release binary downloadable name
release_real_bin="$release_str/$os-$arch/$(basename $package)"
# Release shasum name
release_shasum="$release_str/$os-$arch/$(basename $package).shasum"
# Release sha1sum name
release_shasum="$release_str/$os-$arch/$(basename $package).${release_tag}.shasum"
# Release sha1sum default
release_shasum_default="$release_str/$os-$arch/$(basename $package).shasum"
# Release sha256sum name
release_sha256sum="$release_str/$os-$arch/$(basename $package).${release_tag}.sha256sum"
# Release sha256sum default
release_sha256sum_default="$release_str/$os-$arch/$(basename $package).sha256sum"
# Go build to build the binary.
if [ "${arch}" == "arm" ]; then
# Release binary downloadable name
release_real_bin_6="$release_str/$os-${arch}6vl/$(basename $package)"
CGO_ENABLED=0 GOOS=$os GOARCH=$arch go build --ldflags "${LDFLAGS}" -o $release_bin
release_bin_6="$release_str/$os-${arch}6vl/$(basename $package).$release_tag"
## Support building for ARM6vl
GOARM=6 GOOS=$os GOARCH=$arch go build --ldflags "${LDFLAGS}" -o $release_bin_6
## Copy
$CP -p $release_bin_6 $release_real_bin_6
# Release shasum name
release_shasum_6="$release_str/$os-${arch}6vl/$(basename $package).shasum"
# Calculate shasum
shasum_str=$(${SHASUM} ${release_bin_6})
echo ${shasum_str} | $SED "s/$release_str\/$os-${arch}6vl\///g" > $release_shasum_6
# Release binary downloadable name
release_real_bin_7="$release_str/$os-$arch/$(basename $package)"
release_bin_7="$release_str/$os-$arch/$(basename $package).$release_tag"
## Support building for ARM7vl
GOARM=7 GOOS=$os GOARCH=$arch go build --ldflags "${LDFLAGS}" -o $release_bin_7
## Copy
$CP -p $release_bin_7 $release_real_bin_7
# Release shasum name
release_shasum_7="$release_str/$os-$arch/$(basename $package).shasum"
# Calculate shasum
shasum_str=$(${SHASUM} ${release_bin_7})
echo ${shasum_str} | $SED "s/$release_str\/$os-$arch\///g" > $release_shasum_7
# Create copy
if [ $os == "windows" ]; then
$CP -p $release_bin ${release_real_bin}.exe
else
GOOS=$os GOARCH=$arch go build --ldflags "${LDFLAGS}" -o $release_bin
# Create copy
if [ $os == "windows" ]; then
$CP -p $release_bin ${release_real_bin}.exe
else
$CP -p $release_bin $release_real_bin
fi
# Calculate shasum
shasum_str=$(${SHASUM} ${release_bin})
echo ${shasum_str} | $SED "s/$release_str\/$os-$arch\///g" > $release_shasum
$CP -p $release_bin $release_real_bin
fi
# Calculate sha1sum
shasum_str=$(${SHASUM} ${release_bin})
echo ${shasum_str} | $SED "s/$release_str\/$os-$arch\///g" > $release_shasum
$CP -p $release_shasum $release_shasum_default
# Calculate sha256sum
sha256sum_str=$(${SHA256SUM} ${release_bin})
echo ${sha256sum_str} | $SED "s/$release_str\/$os-$arch\///g" > $release_sha256sum
$CP -p $release_sha256sum $release_sha256sum_default
}
main() {
@@ -111,9 +90,18 @@ main() {
go_build ${each_osarch}
done
else
for each_osarch in $(echo $chosen_osarch | sed 's/,/ /g'); do
go_build ${each_osarch}
local found=0
for each_osarch in ${SUPPORTED_OSARCH}; do
if [ "$chosen_osarch" = "$each_osarch" ]; then
found=1
fi
done
if [ ${found} -eq 1 ]; then
go_build ${chosen_osarch}
else
echo "Unknown architecture \"${chosen_osarch}\""
exit 1
fi
fi
}
+6 -1
View File
@@ -25,6 +25,11 @@ _init() {
OSX_VERSION="10.8"
KNAME=$(uname -s)
ARCH=$(uname -m)
case "${KNAME}" in
SunOS )
ARCH=$(isainfo -k)
;;
esac
}
## FIXME:
@@ -97,7 +102,7 @@ assert_is_supported_arch() {
assert_is_supported_os() {
case "${KNAME}" in
Linux | FreeBSD | OpenBSD | NetBSD | DragonFly )
Linux | FreeBSD | OpenBSD | NetBSD | DragonFly | SunOS )
return
;;
Darwin )
-2
View File
@@ -16,8 +16,6 @@
#
main() {
echo "Checking project is in GOPATH:"
IFS=':' read -r -a paths <<< "$GOPATH"
for path in "${paths[@]}"; do
minio_path="$path/src/github.com/minio/minio"
+81
View File
@@ -0,0 +1,81 @@
#!/bin/sh
#
# Minio Cloud Storage, (C) 2017 Minio, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# If command starts with an option, prepend minio.
if [ "${1}" != "minio" ]; then
if [ -n "${1}" ]; then
set -- minio "$@"
fi
fi
# Wait for all the hosts to come online and have
# their DNS entries populated properly.
docker_wait_hosts() {
hosts="$@"
num_hosts=0
# Count number of hosts in arguments.
for host in $hosts; do
[ $(echo "$host" | grep -E "^http") ] || continue
num_hosts=$((num_hosts+1))
done
if [ $num_hosts -gt 0 ]; then
echo -n "Waiting for all hosts to resolve..."
while true; do
x=0
for host in $hosts; do
[ $(echo "$host" | grep -E "^http") ] || continue
# Extract the domain.
host=$(echo $host | sed -e 's/^http[s]\?:\/\/\([^\/]\+\).*/\1/')
echo -n .
val=$(ping -c 1 $host 2>/dev/null)
if [ $? != 0 ]; then
echo "Failed to lookup $host"
continue
fi
x=$((x+1))
done
# Provided hosts same as successful hosts, should break out.
test $x -eq $num_hosts && break
echo "Failed to resolve hosts.. retrying after 1 second."
sleep 1
done
echo "All hosts are resolving proceeding to initialize Minio."
fi
}
## Look for docker secrets in default documented location.
docker_secrets_env() {
local MINIO_ACCESS_KEY_FILE="/run/secrets/access_key"
local MINIO_SECRET_KEY_FILE="/run/secrets/secret_key"
if [ -f $MINIO_ACCESS_KEY_FILE -a -f $MINIO_SECRET_KEY_FILE ]; then
if [ -f $MINIO_ACCESS_KEY_FILE ]; then
export MINIO_ACCESS_KEY="$(cat "$MINIO_ACCESS_KEY_FILE")"
fi
if [ -f $MINIO_SECRET_KEY_FILE ]; then
export MINIO_SECRET_KEY="$(cat "$MINIO_SECRET_KEY_FILE")"
fi
fi
}
## Set access env from secrets if necessary.
docker_secrets_env
## Wait for all the hosts to come online.
docker_wait_hosts "$@"
exec "$@"
+2 -2
View File
@@ -27,8 +27,8 @@ import (
)
func genLDFlags(version string) string {
var ldflagsStr string
ldflagsStr = "-X github.com/minio/minio/cmd.Version=" + version
ldflagsStr := "-s -w"
ldflagsStr += " -X github.com/minio/minio/cmd.Version=" + version
ldflagsStr += " -X github.com/minio/minio/cmd.ReleaseTag=" + releaseTag(version)
ldflagsStr += " -X github.com/minio/minio/cmd.CommitID=" + commitID()
ldflagsStr += " -X github.com/minio/minio/cmd.ShortCommitID=" + commitID()[:12]
+43
View File
@@ -0,0 +1,43 @@
#!/bin/sh
#
# Minio Cloud Storage, (C) 2017 Minio, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
_init () {
scheme="http://"
address="127.0.0.1:9000"
resource="/minio/index.html"
}
HealthCheckMain () {
# Get the http response code
http_response=$(curl -s -k -o /dev/null -I -w "%{http_code}" ${scheme}${address}${resource})
# Get the http response body
http_response_body=$(curl -k -s ${scheme}${address}${resource})
# server returns response 403 and body "SSL required" if non-TLS connection is attempted on a TLS-configured server.
# change the scheme and try again
if [ "$http_response" = "403" ] && [ "$http_response_body" = "SSL required" ]; then
scheme="https://"
http_response=$(curl -s -k -o /dev/null -I -w "%{http_code}" ${scheme}${address}${resource})
fi
# If http_repsonse is 200 - server is up.
# When MINIO_BROWSER is set to off, curl responds with 404. We assume that the the server is up
[ "$http_response" = "200" ] || [ "$http_response" = "404" ]
}
_init && HealthCheckMain
+408 -71
View File
@@ -17,32 +17,41 @@
package cmd
import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"path"
"strconv"
"sync"
"time"
)
const (
minioAdminOpHeader = "X-Minio-Operation"
minioAdminOpHeader = "X-Minio-Operation"
minioConfigTmpFormat = "config-%s.json"
)
// Type-safe query params.
type mgmtQueryKey string
// Only valid query params for list/clear locks management APIs.
// Only valid query params for mgmt admin APIs.
const (
mgmtBucket mgmtQueryKey = "bucket"
mgmtObject mgmtQueryKey = "object"
mgmtPrefix mgmtQueryKey = "prefix"
mgmtLockDuration mgmtQueryKey = "duration"
mgmtDelimiter mgmtQueryKey = "delimiter"
mgmtMarker mgmtQueryKey = "marker"
mgmtMaxKey mgmtQueryKey = "max-key"
mgmtDryRun mgmtQueryKey = "dry-run"
mgmtBucket mgmtQueryKey = "bucket"
mgmtObject mgmtQueryKey = "object"
mgmtPrefix mgmtQueryKey = "prefix"
mgmtLockDuration mgmtQueryKey = "duration"
mgmtDelimiter mgmtQueryKey = "delimiter"
mgmtMarker mgmtQueryKey = "marker"
mgmtKeyMarker mgmtQueryKey = "key-marker"
mgmtMaxKey mgmtQueryKey = "max-key"
mgmtDryRun mgmtQueryKey = "dry-run"
mgmtUploadIDMarker mgmtQueryKey = "upload-id-marker"
mgmtMaxUploads mgmtQueryKey = "max-uploads"
mgmtUploadID mgmtQueryKey = "upload-id"
)
// ServerVersion - server version
@@ -159,18 +168,12 @@ func (adminAPI adminAPIHandlers) ServiceCredentialsHandler(w http.ResponseWriter
return
}
// Check passed credentials
err = validateAuthKeys(req.Username, req.Password)
creds, err := createCredential(req.Username, req.Password)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
creds := credential{
AccessKey: req.Username,
SecretKey: req.Password,
}
// Notify all other Minio peers to update credentials
updateErrs := updateCredsOnPeers(creds)
for peer, err := range updateErrs {
@@ -205,14 +208,45 @@ type ServerConnStats struct {
Throughput uint64 `json:"throughput,omitempty"`
}
// ServerInfo holds the information that will be returned by ServerInfo API
type ServerInfo struct {
// ServerHTTPMethodStats holds total number of HTTP operations from/to the server,
// including the average duration the call was spent.
type ServerHTTPMethodStats struct {
Count uint64 `json:"count"`
AvgDuration string `json:"avgDuration"`
}
// ServerHTTPStats holds all type of http operations performed to/from the server
// including their average execution time.
type ServerHTTPStats struct {
TotalHEADStats ServerHTTPMethodStats `json:"totalHEADs"`
SuccessHEADStats ServerHTTPMethodStats `json:"successHEADs"`
TotalGETStats ServerHTTPMethodStats `json:"totalGETs"`
SuccessGETStats ServerHTTPMethodStats `json:"successGETs"`
TotalPUTStats ServerHTTPMethodStats `json:"totalPUTs"`
SuccessPUTStats ServerHTTPMethodStats `json:"successPUTs"`
TotalPOSTStats ServerHTTPMethodStats `json:"totalPOSTs"`
SuccessPOSTStats ServerHTTPMethodStats `json:"successPOSTs"`
TotalDELETEStats ServerHTTPMethodStats `json:"totalDELETEs"`
SuccessDELETEStats ServerHTTPMethodStats `json:"successDELETEs"`
}
// ServerInfoData holds storage, connections and other
// information of a given server.
type ServerInfoData struct {
StorageInfo StorageInfo `json:"storage"`
ConnStats ServerConnStats `json:"network"`
HTTPStats ServerHTTPStats `json:"http"`
Properties ServerProperties `json:"server"`
}
// ServerInfoHandler - GET /?server-info
// ServerInfo holds server information result of one node
type ServerInfo struct {
Error string `json:"error"`
Addr string `json:"addr"`
Data *ServerInfoData `json:"data"`
}
// ServerInfoHandler - GET /?info
// ----------
// Get server information
func (adminAPI adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *http.Request) {
@@ -223,58 +257,43 @@ func (adminAPI adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *htt
return
}
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
storage := objLayer.StorageInfo()
// Web service response
reply := make([]ServerInfo, len(globalAdminPeers))
// Build list of enabled ARNs queues
var arns []string
for queueArn := range globalEventNotifier.GetAllExternalTargets() {
arns = append(arns, queueArn)
var wg sync.WaitGroup
// Gather server information for all nodes
for i, p := range globalAdminPeers {
wg.Add(1)
// Gather information from a peer in a goroutine
go func(idx int, peer adminPeer) {
defer wg.Done()
// Initialize server info at index
reply[idx] = ServerInfo{Addr: peer.addr}
serverInfoData, err := peer.cmdRunner.ServerInfoData()
if err != nil {
errorIf(err, "Unable to get server info from %s.", peer.addr)
reply[idx].Error = err.Error()
return
}
reply[idx].Data = &serverInfoData
}(i, p)
}
// Fetch uptimes from all peers. This may fail to due to lack
// of read-quorum availability.
uptime, err := getPeerUptimes(globalAdminPeers)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
errorIf(err, "Unable to get uptime from majority of servers.")
return
}
// Build server properties information
properties := ServerProperties{
Version: Version,
CommitID: CommitID,
Region: serverConfig.GetRegion(),
SQSARN: arns,
Uptime: uptime,
}
// Build network info
connStats := ServerConnStats{
TotalInputBytes: globalConnStats.getTotalInputBytes(),
TotalOutputBytes: globalConnStats.getTotalOutputBytes(),
}
// Build the whole returned information
info := ServerInfo{
StorageInfo: storage,
ConnStats: connStats,
Properties: properties,
}
wg.Wait()
// Marshal API response
jsonBytes, err := json.Marshal(info)
jsonBytes, err := json.Marshal(reply)
if err != nil {
writeErrorResponse(w, ErrInternalError, r.URL)
errorIf(err, "Failed to marshal storage info into json.")
return
}
// Reply with storage information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
@@ -397,8 +416,57 @@ func (adminAPI adminAPIHandlers) ClearLocksHandler(w http.ResponseWriter, r *htt
writeSuccessResponseJSON(w, jsonBytes)
}
// validateHealQueryParams - Validates query params for heal list management API.
func validateHealQueryParams(vars url.Values) (string, string, string, string, int, APIErrorCode) {
// ListUploadsHealHandler - similar to listObjectsHealHandler
// GET
// /?heal&bucket=mybucket&prefix=myprefix&key-marker=mymarker&upload-id-marker=myuploadid&delimiter=mydelimiter&max-uploads=1000
// - bucket is mandatory query parameter
// - rest are optional query parameters List upto maxKey objects that
// need healing in a given bucket matching the given prefix.
func (adminAPI adminAPIHandlers) ListUploadsHealHandler(w http.ResponseWriter, r *http.Request) {
// Get object layer instance.
objLayer := newObjectLayerFn()
if objLayer == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
// Validate request signature.
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// Validate query params.
vars := r.URL.Query()
bucket := vars.Get(string(mgmtBucket))
prefix, keyMarker, uploadIDMarker, delimiter, maxUploads, _ := getBucketMultipartResources(r.URL.Query())
if err := checkListMultipartArgs(bucket, prefix, keyMarker, uploadIDMarker, delimiter, objLayer); err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
if maxUploads <= 0 || maxUploads > maxUploadsList {
writeErrorResponse(w, ErrInvalidMaxUploads, r.URL)
return
}
// Get the list objects to be healed.
listMultipartInfos, err := objLayer.ListUploadsHeal(bucket, prefix,
keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
listResponse := generateListMultipartUploadsResponse(bucket, listMultipartInfos)
// Write success response.
writeSuccessResponseXML(w, encodeResponse(listResponse))
}
// extractListObjectsHealQuery - Validates query params for heal objects list management API.
func extractListObjectsHealQuery(vars url.Values) (string, string, string, string, int, APIErrorCode) {
bucket := vars.Get(string(mgmtBucket))
prefix := vars.Get(string(mgmtPrefix))
marker := vars.Get(string(mgmtMarker))
@@ -415,10 +483,13 @@ func validateHealQueryParams(vars url.Values) (string, string, string, string, i
return "", "", "", "", 0, ErrInvalidObjectName
}
// check if maxKey is a valid integer.
maxKey, err := strconv.Atoi(maxKeyStr)
if err != nil {
return "", "", "", "", 0, ErrInvalidMaxKeys
// check if maxKey is a valid integer, if present.
var maxKey int
var err error
if maxKeyStr != "" {
if maxKey, err = strconv.Atoi(maxKeyStr); err != nil {
return "", "", "", "", 0, ErrInvalidMaxKeys
}
}
// Validate prefix, marker, delimiter and maxKey.
@@ -451,7 +522,7 @@ func (adminAPI adminAPIHandlers) ListObjectsHealHandler(w http.ResponseWriter, r
// Validate query params.
vars := r.URL.Query()
bucket, prefix, marker, delimiter, maxKey, adminAPIErr := validateHealQueryParams(vars)
bucket, prefix, marker, delimiter, maxKey, adminAPIErr := extractListObjectsHealQuery(vars)
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
@@ -550,6 +621,42 @@ func isDryRun(qval url.Values) bool {
return false
}
// healResult - represents result of a heal operation like
// heal-object, heal-upload.
type healResult struct {
State healState `json:"state"`
}
// healState - different states of heal operation
type healState int
const (
// healNone - none of the disks healed
healNone healState = iota
// healPartial - some disks were healed, others were offline
healPartial
// healOK - all disks were healed
healOK
)
// newHealResult - returns healResult given number of disks healed and
// number of disks offline
func newHealResult(numHealedDisks, numOfflineDisks int) healResult {
var state healState
switch {
case numHealedDisks == 0:
state = healNone
case numOfflineDisks > 0:
state = healPartial
default:
state = healOK
}
return healResult{State: state}
}
// HealObjectHandler - POST /?heal&bucket=mybucket&object=myobject&dry-run
// - x-minio-operation = object
// - bucket and object are both mandatory query parameters
@@ -592,14 +699,95 @@ func (adminAPI adminAPIHandlers) HealObjectHandler(w http.ResponseWriter, r *htt
return
}
err := objLayer.HealObject(bucket, object)
numOfflineDisks, numHealedDisks, err := objLayer.HealObject(bucket, object)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
jsonBytes, err := json.Marshal(newHealResult(numHealedDisks, numOfflineDisks))
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Return 200 on success.
writeSuccessResponseHeadersOnly(w)
writeSuccessResponseJSON(w, jsonBytes)
}
// HealUploadHandler - POST /?heal&bucket=mybucket&object=myobject&upload-id=myuploadID&dry-run
// - x-minio-operation = upload
// - bucket, object and upload-id are mandatory query parameters
// Heal a given upload, if present.
func (adminAPI adminAPIHandlers) HealUploadHandler(w http.ResponseWriter, r *http.Request) {
// Get object layer instance.
objLayer := newObjectLayerFn()
if objLayer == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
// Validate request signature.
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
vars := r.URL.Query()
bucket := vars.Get(string(mgmtBucket))
object := vars.Get(string(mgmtObject))
uploadID := vars.Get(string(mgmtUploadID))
uploadObj := path.Join(bucket, object, uploadID)
// Validate bucket and object names as supplied via query
// parameters.
if err := checkBucketAndObjectNames(bucket, object); err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Validate the bucket and object w.r.t backend representation
// of an upload.
if err := checkBucketAndObjectNames(minioMetaMultipartBucket,
uploadObj); err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Check if upload exists.
if _, err := objLayer.GetObjectInfo(minioMetaMultipartBucket,
uploadObj); err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// if dry-run is set in query params then perform validations
// and return success.
if isDryRun(vars) {
writeSuccessResponseHeadersOnly(w)
return
}
//We are able to use HealObject for healing an upload since an
//ongoing upload has the same backend representation as an
//object. The 'object' corresponding to a given bucket,
//object and uploadID is
//.minio.sys/multipart/bucket/object/uploadID.
numOfflineDisks, numHealedDisks, err := objLayer.HealObject(minioMetaMultipartBucket, uploadObj)
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
jsonBytes, err := json.Marshal(newHealResult(numHealedDisks, numOfflineDisks))
if err != nil {
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Return 200 on success.
writeSuccessResponseJSON(w, jsonBytes)
}
// HealFormatHandler - POST /?heal&dry-run
@@ -672,3 +860,152 @@ func (adminAPI adminAPIHandlers) HealFormatHandler(w http.ResponseWriter, r *htt
// Return 200 on success.
writeSuccessResponseHeadersOnly(w)
}
// GetConfigHandler - GET /?config
// - x-minio-operation = get
// Get config.json of this minio setup.
func (adminAPI adminAPIHandlers) GetConfigHandler(w http.ResponseWriter, r *http.Request) {
// Validate request signature.
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// check if objectLayer is initialized, if not return.
if newObjectLayerFn() == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
// Get config.json from all nodes. In a single node setup, it
// returns local config.json.
configBytes, err := getPeerConfig(globalAdminPeers)
if err != nil {
errorIf(err, "Failed to get config from peers")
writeErrorResponse(w, toAdminAPIErrCode(err), r.URL)
return
}
writeSuccessResponseJSON(w, configBytes)
}
// toAdminAPIErrCode - converts errXLWriteQuorum error to admin API
// specific error.
func toAdminAPIErrCode(err error) APIErrorCode {
switch err {
case errXLWriteQuorum:
return ErrAdminConfigNoQuorum
}
return toAPIErrorCode(err)
}
// SetConfigResult - represents detailed results of a set-config
// operation.
type nodeSummary struct {
Name string `json:"name"`
ErrSet bool `json:"errSet"`
ErrMsg string `json:"errMsg"`
}
type setConfigResult struct {
NodeResults []nodeSummary `json:"nodeResults"`
Status bool `json:"status"`
}
// writeSetConfigResponse - writes setConfigResult value as json depending on the status.
func writeSetConfigResponse(w http.ResponseWriter, peers adminPeers, errs []error, status bool, reqURL *url.URL) {
var nodeResults []nodeSummary
// Build nodeResults based on error values received during
// set-config operation.
for i := range errs {
nodeResults = append(nodeResults, nodeSummary{
Name: peers[i].addr,
ErrSet: errs[i] != nil,
ErrMsg: fmt.Sprintf("%v", errs[i]),
})
}
result := setConfigResult{
Status: status,
NodeResults: nodeResults,
}
// The following elaborate json encoding is to avoid escaping
// '<', '>' in <nil>. Note: json.Encoder.Encode() adds a
// gratuitous "\n".
var resultBuf bytes.Buffer
enc := json.NewEncoder(&resultBuf)
enc.SetEscapeHTML(false)
jsonErr := enc.Encode(result)
if jsonErr != nil {
writeErrorResponse(w, toAPIErrorCode(jsonErr), reqURL)
return
}
writeSuccessResponseJSON(w, resultBuf.Bytes())
return
}
// SetConfigHandler - PUT /?config
// - x-minio-operation = set
func (adminAPI adminAPIHandlers) SetConfigHandler(w http.ResponseWriter, r *http.Request) {
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
writeErrorResponse(w, ErrServerNotInitialized, r.URL)
return
}
// Validate request signature.
adminAPIErr := checkRequestAuthType(r, "", "", "")
if adminAPIErr != ErrNone {
writeErrorResponse(w, adminAPIErr, r.URL)
return
}
// Read configuration bytes from request body.
configBytes, err := ioutil.ReadAll(r.Body)
if err != nil {
errorIf(err, "Failed to read config from request body.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
// Write config received from request onto a temporary file on
// all nodes.
tmpFileName := fmt.Sprintf(minioConfigTmpFormat, mustGetUUID())
errs := writeTmpConfigPeers(globalAdminPeers, tmpFileName, configBytes)
// Check if the operation succeeded in quorum or more nodes.
rErr := reduceWriteQuorumErrs(errs, nil, len(globalAdminPeers)/2+1)
if rErr != nil {
writeSetConfigResponse(w, globalAdminPeers, errs, false, r.URL)
return
}
// Take a lock on minio/config.json. NB minio is a reserved
// bucket name and wouldn't conflict with normal object
// operations.
configLock := globalNSMutex.NewNSLock(minioReservedBucket, minioConfigFile)
configLock.Lock()
defer configLock.Unlock()
// Rename the temporary config file to config.json
errs = commitConfigPeers(globalAdminPeers, tmpFileName)
rErr = reduceWriteQuorumErrs(errs, nil, len(globalAdminPeers)/2+1)
if rErr != nil {
writeSetConfigResponse(w, globalAdminPeers, errs, false, r.URL)
return
}
// serverMux (cmd/server-mux.go) implements graceful shutdown,
// where all listeners are closed and process restart/shutdown
// happens after 5s or completion of all ongoing http
// requests, whichever is earlier.
writeSetConfigResponse(w, globalAdminPeers, errs, true, r.URL)
// Restart all node for the modified config to take effect.
sendServiceCmd(globalAdminPeers, serviceRestart)
}
+690 -54
View File
@@ -20,16 +20,115 @@ import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/http/httptest"
"net/url"
"reflect"
"testing"
"time"
router "github.com/gorilla/mux"
)
var configJSON = []byte(`{
"version": "13",
"credential": {
"accessKey": "minio",
"secretKey": "minio123"
},
"region": "us-west-1",
"logger": {
"console": {
"enable": true,
"level": "fatal"
},
"file": {
"enable": false,
"fileName": "",
"level": ""
}
},
"notify": {
"amqp": {
"1": {
"enable": false,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"streaming": {
"enable": false,
"clusterID": "",
"clientID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"elasticsearch": {
"1": {
"enable": false,
"url": "",
"index": ""
}
},
"redis": {
"1": {
"enable": false,
"address": "",
"password": "",
"key": ""
}
},
"postgresql": {
"1": {
"enable": false,
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": ""
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
}
}
}
}`)
// adminXLTestBed - encapsulates subsystems that need to be setup for
// admin-handler unit tests.
type adminXLTestBed struct {
@@ -57,14 +156,9 @@ func prepareAdminXLTestBed() (*adminXLTestBed, error) {
}
// Initialize boot time
globalBootTime = time.Now().UTC()
globalBootTime = UTCNow()
// Set globalEndpoints for a single node XL setup.
for _, xlDir := range xlDirs {
globalEndpoints = append(globalEndpoints, &url.URL{
Path: xlDir,
})
}
globalEndpoints = mustGetNewEndpointList(xlDirs...)
// Set globalIsXL to indicate that the setup uses an erasure code backend.
globalIsXL = true
@@ -202,14 +296,8 @@ func testServicesCmdHandler(cmd cmdType, t *testing.T) {
// Initialize admin peers to make admin RPC calls. Note: In a
// single node setup, this degenerates to a simple function
// call under the hood.
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
// Set globalMinioAddr to be able to distinguish local endpoints from remote.
globalMinioAddr = eps[0].Host
initGlobalAdminPeers(eps)
globalMinioAddr = "127.0.0.1:9000"
initGlobalAdminPeers(mustGetNewEndpointList("http://127.0.0.1:9000/d1"))
// Setting up a go routine to simulate ServerMux's
// handleServiceSignals for stop and restart commands.
@@ -268,14 +356,8 @@ func TestServiceSetCreds(t *testing.T) {
// Initialize admin peers to make admin RPC calls. Note: In a
// single node setup, this degenerates to a simple function
// call under the hood.
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
// Set globalMinioAddr to be able to distinguish local endpoints from remote.
globalMinioAddr = eps[0].Host
initGlobalAdminPeers(eps)
globalMinioAddr = "127.0.0.1:9000"
initGlobalAdminPeers(mustGetNewEndpointList("http://127.0.0.1:9000/d1"))
credentials := serverConfig.GetCredential()
var body []byte
@@ -356,14 +438,8 @@ func TestListLocksHandler(t *testing.T) {
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
// Set globalMinioAddr to be able to distinguish local endpoints from remote.
globalMinioAddr = eps[0].Host
initGlobalAdminPeers(eps)
globalMinioAddr = "127.0.0.1:9000"
initGlobalAdminPeers(mustGetNewEndpointList("http://127.0.0.1:9000/d1"))
testCases := []struct {
bucket string
@@ -431,11 +507,7 @@ func TestClearLocksHandler(t *testing.T) {
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
eps, err := parseStorageEndpoints([]string{"http://127.0.0.1"})
if err != nil {
t.Fatalf("Failed to parse storage end point - %v", err)
}
initGlobalAdminPeers(eps)
initGlobalAdminPeers(mustGetNewEndpointList("http://127.0.0.1:9000/d1"))
testCases := []struct {
bucket string
@@ -643,7 +715,7 @@ func TestValidateHealQueryParams(t *testing.T) {
}
for i, test := range testCases {
vars := mkListObjectsQueryVal(test.bucket, test.prefix, test.marker, test.delimiter, test.maxKeys)
_, _, _, _, _, actualErr := validateHealQueryParams(vars)
_, _, _, _, _, actualErr := extractListObjectsHealQuery(vars)
if actualErr != test.apiErr {
t.Errorf("Test %d - Expected %v but received %v",
i+1, getAPIError(test.apiErr), getAPIError(actualErr))
@@ -659,7 +731,7 @@ func TestListObjectsHealHandler(t *testing.T) {
}
defer adminTestBed.TearDown()
err = adminTestBed.objLayer.MakeBucket("mybucket")
err = adminTestBed.objLayer.MakeBucketWithLocation("mybucket", "")
if err != nil {
t.Fatalf("Failed to make bucket - %v", err)
}
@@ -759,9 +831,6 @@ func TestListObjectsHealHandler(t *testing.T) {
}
for i, test := range testCases {
if i != 0 {
continue
}
queryVal := mkListObjectsQueryVal(test.bucket, test.prefix, test.marker, test.delimiter, test.maxKeys)
req, err := newTestRequest("GET", "/?"+queryVal.Encode(), 0, nil)
if err != nil {
@@ -790,7 +859,7 @@ func TestHealBucketHandler(t *testing.T) {
}
defer adminTestBed.TearDown()
err = adminTestBed.objLayer.MakeBucket("mybucket")
err = adminTestBed.objLayer.MakeBucketWithLocation("mybucket", "")
if err != nil {
t.Fatalf("Failed to make bucket - %v", err)
}
@@ -867,7 +936,7 @@ func TestHealObjectHandler(t *testing.T) {
// Create an object myobject under bucket mybucket.
bucketName := "mybucket"
objName := "myobject"
err = adminTestBed.objLayer.MakeBucket(bucketName)
err = adminTestBed.objLayer.MakeBucketWithLocation(bucketName, "")
if err != nil {
t.Fatalf("Failed to make bucket %s - %v", bucketName, err)
}
@@ -954,6 +1023,160 @@ func TestHealObjectHandler(t *testing.T) {
t.Errorf("Test %d - Expected HTTP status code %d but received %d", i+1, test.statusCode, rec.Code)
}
}
}
// buildAdminRequest - helper function to build an admin API request.
func buildAdminRequest(queryVal url.Values, opHdr, method string,
contentLength int64, bodySeeker io.ReadSeeker) (*http.Request, error) {
req, err := newTestRequest(method, "/?"+queryVal.Encode(), contentLength, bodySeeker)
if err != nil {
return nil, traceError(err)
}
req.Header.Set(minioAdminOpHeader, opHdr)
cred := serverConfig.GetCredential()
err = signRequestV4(req, cred.AccessKey, cred.SecretKey)
if err != nil {
return nil, traceError(err)
}
return req, nil
}
// mkHealUploadQuery - helper to build HealUploadHandler query string.
func mkHealUploadQuery(bucket, object, uploadID, dryRun string) url.Values {
queryVal := url.Values{}
queryVal.Set(string(mgmtBucket), bucket)
queryVal.Set(string(mgmtObject), object)
queryVal.Set(string(mgmtUploadID), uploadID)
queryVal.Set("heal", "")
queryVal.Set(string(mgmtDryRun), dryRun)
return queryVal
}
// TestHealUploadHandler - test for HealUploadHandler.
func TestHealUploadHandler(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
// Create an object myobject under bucket mybucket.
bucketName := "mybucket"
objName := "myobject"
err = adminTestBed.objLayer.MakeBucketWithLocation(bucketName, "")
if err != nil {
t.Fatalf("Failed to make bucket %s - %v", bucketName, err)
}
// Create a new multipart upload.
uploadID, err := adminTestBed.objLayer.NewMultipartUpload(bucketName, objName, nil)
if err != nil {
t.Fatalf("Failed to create a new multipart upload %s/%s - %v",
bucketName, objName, err)
}
// Upload a part.
partID := 1
_, err = adminTestBed.objLayer.PutObjectPart(bucketName, objName, uploadID,
partID, int64(len("hello")), bytes.NewReader([]byte("hello")), "", "")
if err != nil {
t.Fatalf("Failed to upload part %d of %s/%s - %v", partID,
bucketName, objName, err)
}
testCases := []struct {
bucket string
object string
dryrun string
statusCode int
}{
// 1. Valid test case.
{
bucket: bucketName,
object: objName,
statusCode: http.StatusOK,
},
// 2. Invalid bucket name.
{
bucket: `invalid\\Bucket`,
object: "myobject",
statusCode: http.StatusBadRequest,
},
// 3. Bucket not found.
{
bucket: "bucketnotfound",
object: "myobject",
statusCode: http.StatusNotFound,
},
// 4. Invalid object name.
{
bucket: bucketName,
object: `invalid\\Object`,
statusCode: http.StatusBadRequest,
},
// 5. Object not found.
{
bucket: bucketName,
object: "objectnotfound",
statusCode: http.StatusNotFound,
},
// 6. Valid test case with dry-run.
{
bucket: bucketName,
object: objName,
dryrun: "yes",
statusCode: http.StatusOK,
},
}
for i, test := range testCases {
// Prepare query params.
queryVal := mkHealUploadQuery(test.bucket, test.object, uploadID, test.dryrun)
req, err1 := buildAdminRequest(queryVal, "upload", http.MethodPost, 0, nil)
if err1 != nil {
t.Fatalf("Test %d - Failed to construct heal object request - %v", i+1, err1)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if test.statusCode != rec.Code {
t.Errorf("Test %d - Expected HTTP status code %d but received %d", i+1, test.statusCode, rec.Code)
}
}
sample := testCases[0]
// Modify authorization header after signing to test signature
// mismatch handling.
queryVal := mkHealUploadQuery(sample.bucket, sample.object, uploadID, sample.dryrun)
req, err := buildAdminRequest(queryVal, "upload", "POST", 0, nil)
if err != nil {
t.Fatalf("Failed to construct heal object request - %v", err)
}
// Set x-amz-date to a date different than time of signing.
req.Header.Set("x-amz-date", time.Time{}.Format(iso8601Format))
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusForbidden {
t.Errorf("Expected %d but received %d", http.StatusBadRequest, rec.Code)
}
// Set objectAPI to nil to test Server not initialized case.
resetGlobalObjectAPI()
queryVal = mkHealUploadQuery(sample.bucket, sample.object, uploadID, sample.dryrun)
req, err = buildAdminRequest(queryVal, "upload", "POST", 0, nil)
if err != nil {
t.Fatalf("Failed to construct heal object request - %v", err)
}
rec = httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("Expected %d but received %d", http.StatusServiceUnavailable, rec.Code)
}
}
// TestHealFormatHandler - test for HealFormatHandler.
@@ -967,24 +1190,437 @@ func TestHealFormatHandler(t *testing.T) {
// Prepare query params for heal-format mgmt REST API.
queryVal := url.Values{}
queryVal.Set("heal", "")
req, err := newTestRequest("POST", "/?"+queryVal.Encode(), 0, nil)
req, err := buildAdminRequest(queryVal, "format", "POST", 0, nil)
if err != nil {
t.Fatalf("Failed to construct heal object request - %v", err)
}
// Set x-minio-operation header to format.
req.Header.Set(minioAdminOpHeader, "format")
// Sign the request using signature v4.
cred := serverConfig.GetCredential()
err = signRequestV4(req, cred.AccessKey, cred.SecretKey)
if err != nil {
t.Fatalf("Failed to sign heal object request - %v", err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("Expected to succeed but failed with %d", rec.Code)
}
}
// TestGetConfigHandler - test for GetConfigHandler.
func TestGetConfigHandler(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
globalMinioAddr = "127.0.0.1:9000"
initGlobalAdminPeers(mustGetNewEndpointList("http://127.0.0.1:9000/d1"))
// Prepare query params for get-config mgmt REST API.
queryVal := url.Values{}
queryVal.Set("config", "")
req, err := buildAdminRequest(queryVal, "get", http.MethodGet, 0, nil)
if err != nil {
t.Fatalf("Failed to construct get-config object request - %v", err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("Expected to succeed but failed with %d", rec.Code)
}
}
// TestSetConfigHandler - test for SetConfigHandler.
func TestSetConfigHandler(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
globalMinioAddr = "127.0.0.1:9000"
initGlobalAdminPeers(mustGetNewEndpointList("http://127.0.0.1:9000/d1"))
// SetConfigHandler restarts minio setup - need to start a
// signal receiver to receive on globalServiceSignalCh.
go testServiceSignalReceiver(restartCmd, t)
// Prepare query params for set-config mgmt REST API.
queryVal := url.Values{}
queryVal.Set("config", "")
req, err := buildAdminRequest(queryVal, "set", http.MethodPut, int64(len(configJSON)),
bytes.NewReader(configJSON))
if err != nil {
t.Fatalf("Failed to construct get-config object request - %v", err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("Expected to succeed but failed with %d", rec.Code)
}
result := setConfigResult{}
err = json.NewDecoder(rec.Body).Decode(&result)
if err != nil {
t.Fatalf("Failed to decode set config result json %v", err)
}
if !result.Status {
t.Error("Expected set-config to succeed, but failed")
}
}
func TestAdminServerInfo(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
// Initialize admin peers to make admin RPC calls.
globalMinioAddr = "127.0.0.1:9000"
initGlobalAdminPeers(mustGetNewEndpointList("http://127.0.0.1:9000/d1"))
// Prepare query params for set-config mgmt REST API.
queryVal := url.Values{}
queryVal.Set("info", "")
req, err := buildAdminRequest(queryVal, "", http.MethodGet, 0, nil)
if err != nil {
t.Fatalf("Failed to construct get-config object request - %v", err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("Expected to succeed but failed with %d", rec.Code)
}
results := []ServerInfo{}
err = json.NewDecoder(rec.Body).Decode(&results)
if err != nil {
t.Fatalf("Failed to decode set config result json %v", err)
}
if len(results) == 0 {
t.Error("Expected at least one server info result")
}
for _, serverInfo := range results {
if len(serverInfo.Addr) == 0 {
t.Error("Expected server address to be non empty")
}
if serverInfo.Error != "" {
t.Errorf("Unexpected error = %v\n", serverInfo.Error)
}
if serverInfo.Data.StorageInfo.Free == 0 {
t.Error("Expected StorageInfo.Free to be non empty")
}
if serverInfo.Data.Properties.Region != globalMinioDefaultRegion {
t.Errorf("Expected %s, got %s", globalMinioDefaultRegion, serverInfo.Data.Properties.Region)
}
}
}
// TestToAdminAPIErr - test for toAdminAPIErr helper function.
func TestToAdminAPIErr(t *testing.T) {
testCases := []struct {
err error
expectedAPIErr APIErrorCode
}{
// 1. Server not in quorum.
{
err: errXLWriteQuorum,
expectedAPIErr: ErrAdminConfigNoQuorum,
},
// 2. No error.
{
err: nil,
expectedAPIErr: ErrNone,
},
// 3. Non-admin API specific error.
{
err: errDiskNotFound,
expectedAPIErr: toAPIErrorCode(errDiskNotFound),
},
}
for i, test := range testCases {
actualErr := toAdminAPIErrCode(test.err)
if actualErr != test.expectedAPIErr {
t.Errorf("Test %d: Expected %v but received %v",
i+1, test.expectedAPIErr, actualErr)
}
}
}
func TestWriteSetConfigResponse(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatal(err)
}
defer removeAll(rootPath)
testCases := []struct {
status bool
errs []error
}{
// 1. all nodes returned success.
{
status: true,
errs: []error{nil, nil, nil, nil},
},
// 2. some nodes returned errors.
{
status: false,
errs: []error{errDiskNotFound, nil, errDiskAccessDenied, errFaultyDisk},
},
}
testPeers := []adminPeer{
{
addr: "localhost:9001",
},
{
addr: "localhost:9002",
},
{
addr: "localhost:9003",
},
{
addr: "localhost:9004",
},
}
testURL, err := url.Parse("http://dummy.com")
if err != nil {
t.Fatalf("Failed to parse a place-holder url")
}
var actualResult setConfigResult
for i, test := range testCases {
rec := httptest.NewRecorder()
writeSetConfigResponse(rec, testPeers, test.errs, test.status, testURL)
resp := rec.Result()
jsonBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("Test %d: Failed to read response %v", i+1, err)
}
err = json.Unmarshal(jsonBytes, &actualResult)
if err != nil {
t.Fatalf("Test %d: Failed to unmarshal json %v", i+1, err)
}
if actualResult.Status != test.status {
t.Errorf("Test %d: Expected status %v but received %v", i+1, test.status, actualResult.Status)
}
for p, res := range actualResult.NodeResults {
if res.Name != testPeers[p].addr {
t.Errorf("Test %d: Expected node name %s but received %s", i+1, testPeers[p].addr, res.Name)
}
expectedErrMsg := fmt.Sprintf("%v", test.errs[p])
if res.ErrMsg != expectedErrMsg {
t.Errorf("Test %d: Expected error %s but received %s", i+1, expectedErrMsg, res.ErrMsg)
}
expectedErrSet := test.errs[p] != nil
if res.ErrSet != expectedErrSet {
t.Errorf("Test %d: Expected ErrSet %v but received %v", i+1, expectedErrSet, res.ErrSet)
}
}
}
}
// mkListUploadsHealQuery - helper function to construct query values for
// listUploadsHeal.
func mkListUploadsHealQuery(bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploadsStr string) url.Values {
queryVal := make(url.Values)
queryVal.Set("heal", "")
queryVal.Set(string(mgmtBucket), bucket)
queryVal.Set(string(mgmtPrefix), prefix)
queryVal.Set(string(mgmtKeyMarker), keyMarker)
queryVal.Set(string(mgmtUploadIDMarker), uploadIDMarker)
queryVal.Set(string(mgmtDelimiter), delimiter)
queryVal.Set(string(mgmtMaxUploads), maxUploadsStr)
return queryVal
}
func TestListHealUploadsHandler(t *testing.T) {
adminTestBed, err := prepareAdminXLTestBed()
if err != nil {
t.Fatal("Failed to initialize a single node XL backend for admin handler tests.")
}
defer adminTestBed.TearDown()
err = adminTestBed.objLayer.MakeBucketWithLocation("mybucket", "")
if err != nil {
t.Fatalf("Failed to make bucket - %v", err)
}
// Delete bucket after running all test cases.
defer adminTestBed.objLayer.DeleteBucket("mybucket")
testCases := []struct {
bucket string
prefix string
keyMarker string
delimiter string
maxKeys string
statusCode int
expectedResp ListMultipartUploadsResponse
}{
// 1. Valid params.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "10",
statusCode: http.StatusOK,
expectedResp: ListMultipartUploadsResponse{
XMLName: xml.Name{Space: "http://s3.amazonaws.com/doc/2006-03-01/", Local: "ListMultipartUploadsResult"},
Bucket: "mybucket",
KeyMarker: "prefix11",
Delimiter: "/",
Prefix: "prefix",
MaxUploads: 10,
},
},
// 2. Valid params with empty prefix.
{
bucket: "mybucket",
prefix: "",
keyMarker: "",
delimiter: "/",
maxKeys: "10",
statusCode: http.StatusOK,
expectedResp: ListMultipartUploadsResponse{
XMLName: xml.Name{Space: "http://s3.amazonaws.com/doc/2006-03-01/", Local: "ListMultipartUploadsResult"},
Bucket: "mybucket",
KeyMarker: "",
Delimiter: "/",
Prefix: "",
MaxUploads: 10,
},
},
// 3. Invalid params with invalid bucket.
{
bucket: `invalid\\Bucket`,
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "10",
statusCode: getAPIError(ErrInvalidBucketName).HTTPStatusCode,
},
// 4. Invalid params with invalid prefix.
{
bucket: "mybucket",
prefix: `invalid\\Prefix`,
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "10",
statusCode: getAPIError(ErrInvalidObjectName).HTTPStatusCode,
},
// 5. Invalid params with invalid maxKeys.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "-1",
statusCode: getAPIError(ErrInvalidMaxUploads).HTTPStatusCode,
},
// 6. Invalid params with unsupported prefix marker combination.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "notmatchingmarker",
delimiter: "/",
maxKeys: "10",
statusCode: getAPIError(ErrNotImplemented).HTTPStatusCode,
},
// 7. Invalid params with unsupported delimiter.
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "notmatchingmarker",
delimiter: "unsupported",
maxKeys: "10",
statusCode: getAPIError(ErrNotImplemented).HTTPStatusCode,
},
// 8. Invalid params with invalid max Keys
{
bucket: "mybucket",
prefix: "prefix",
keyMarker: "prefix11",
delimiter: "/",
maxKeys: "999999999999999999999999999",
statusCode: getAPIError(ErrInvalidMaxUploads).HTTPStatusCode,
},
}
for i, test := range testCases {
queryVal := mkListUploadsHealQuery(test.bucket, test.prefix, test.keyMarker, "", test.delimiter, test.maxKeys)
req, err := buildAdminRequest(queryVal, "list-uploads", http.MethodGet, 0, nil)
if err != nil {
t.Fatalf("Test %d - Failed to construct list uploads needing heal request - %v", i+1, err)
}
rec := httptest.NewRecorder()
adminTestBed.mux.ServeHTTP(rec, req)
if test.statusCode != rec.Code {
t.Errorf("Test %d - Expected HTTP status code %d but received %d", i+1, test.statusCode, rec.Code)
}
// Compare result with the expected one only when we receive 200 OK
if rec.Code == http.StatusOK {
resp := rec.Result()
xmlBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Errorf("Test %d: Failed to read response %v", i+1, err)
}
var actualResult ListMultipartUploadsResponse
err = xml.Unmarshal(xmlBytes, &actualResult)
if err != nil {
t.Errorf("Test %d: Failed to unmarshal xml %v", i+1, err)
}
if !reflect.DeepEqual(test.expectedResp, actualResult) {
t.Fatalf("Test %d: Unexpected response `%+v`, expected: `%+v`", i+1, test.expectedResp, actualResult)
}
}
}
}
// Test for newHealResult helper function.
func TestNewHealResult(t *testing.T) {
testCases := []struct {
healedDisks int
offlineDisks int
state healState
}{
// 1. No disks healed, no disks offline.
{0, 0, healNone},
// 2. No disks healed, non-zero disks offline.
{0, 1, healNone},
// 3. Non-zero disks healed, no disks offline.
{1, 0, healOK},
// 4. Non-zero disks healed, non-zero disks offline.
{1, 1, healPartial},
}
for i, test := range testCases {
actual := newHealResult(test.healedDisks, test.offlineDisks)
if actual.State != test.state {
t.Errorf("Test %d: Expected %v but received %v", i+1,
test.state, actual.State)
}
}
}
+11
View File
@@ -53,6 +53,8 @@ func registerAdminRouter(mux *router.Router) {
// List Objects needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-objects").HandlerFunc(adminAPI.ListObjectsHealHandler)
// List Uploads needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-uploads").HandlerFunc(adminAPI.ListUploadsHealHandler)
// List Buckets needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-buckets").HandlerFunc(adminAPI.ListBucketsHealHandler)
@@ -62,4 +64,13 @@ func registerAdminRouter(mux *router.Router) {
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "object").HandlerFunc(adminAPI.HealObjectHandler)
// Heal Format.
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "format").HandlerFunc(adminAPI.HealFormatHandler)
// Heal Uploads.
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "upload").HandlerFunc(adminAPI.HealUploadHandler)
/// Config operations
// Get config
adminRouter.Methods("GET").Queries("config", "").Headers(minioAdminOpHeader, "get").HandlerFunc(adminAPI.GetConfigHandler)
// Set Config
adminRouter.Methods("PUT").Queries("config", "").Headers(minioAdminOpHeader, "set").HandlerFunc(adminAPI.SetConfigHandler)
}
+340 -53
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2014-2016 Minio, Inc.
* Minio Cloud Storage, (C) 2014, 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,11 +17,30 @@
package cmd
import (
"net/url"
"encoding/json"
"errors"
"fmt"
"net"
"os"
"path"
"path/filepath"
"reflect"
"sort"
"sync"
"time"
"github.com/minio/minio-go/pkg/set"
)
const (
// Admin service names
serviceRestartRPC = "Admin.Restart"
listLocksRPC = "Admin.ListLocks"
reInitDisksRPC = "Admin.ReInitDisks"
serverInfoDataRPC = "Admin.ServerInfoData"
getConfigRPC = "Admin.GetConfig"
writeTmpConfigRPC = "Admin.WriteTmpConfig"
commitConfigRPC = "Admin.CommitConfig"
)
// localAdminClient - represents admin operation to be executed locally.
@@ -40,7 +59,10 @@ type adminCmdRunner interface {
Restart() error
ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error)
ReInitDisks() error
Uptime() (time.Duration, error)
ServerInfoData() (ServerInfoData, error)
GetConfig() ([]byte, error)
WriteTmpConfig(tmpFileName string, configBytes []byte) error
CommitConfig(tmpFileName string) error
}
// Restart - Sends a message over channel to the go-routine
@@ -59,7 +81,7 @@ func (lc localAdminClient) ListLocks(bucket, prefix string, duration time.Durati
func (rc remoteAdminClient) Restart() error {
args := AuthRPCArgs{}
reply := AuthRPCReply{}
return rc.Call("Admin.Restart", &args, &reply)
return rc.Call(serviceRestartRPC, &args, &reply)
}
// ListLocks - Sends list locks command to remote server via RPC.
@@ -70,7 +92,7 @@ func (rc remoteAdminClient) ListLocks(bucket, prefix string, duration time.Durat
duration: duration,
}
var reply ListLocksReply
if err := rc.Call("Admin.ListLocks", &listArgs, &reply); err != nil {
if err := rc.Call(listLocksRPC, &listArgs, &reply); err != nil {
return nil, err
}
return reply.volLocks, nil
@@ -87,29 +109,120 @@ func (lc localAdminClient) ReInitDisks() error {
func (rc remoteAdminClient) ReInitDisks() error {
args := AuthRPCArgs{}
reply := AuthRPCReply{}
return rc.Call("Admin.ReInitDisks", &args, &reply)
return rc.Call(reInitDisksRPC, &args, &reply)
}
// Uptime - Returns the uptime of this server. Timestamp is taken
// after object layer is initialized.
func (lc localAdminClient) Uptime() (time.Duration, error) {
// ServerInfoData - Returns the server info of this server.
func (lc localAdminClient) ServerInfoData() (sid ServerInfoData, e error) {
if globalBootTime.IsZero() {
return time.Duration(0), errServerNotInitialized
return sid, errServerNotInitialized
}
return time.Now().UTC().Sub(globalBootTime), nil
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
return sid, errServerNotInitialized
}
storage := objLayer.StorageInfo()
var arns []string
for queueArn := range globalEventNotifier.GetAllExternalTargets() {
arns = append(arns, queueArn)
}
return ServerInfoData{
StorageInfo: storage,
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
SQSARN: arns,
Region: serverConfig.GetRegion(),
},
}, nil
}
// Uptime - returns the uptime of the server to which the RPC call is made.
func (rc remoteAdminClient) Uptime() (time.Duration, error) {
// ServerInfo - returns the server info of the server to which the RPC call is made.
func (rc remoteAdminClient) ServerInfoData() (sid ServerInfoData, e error) {
args := AuthRPCArgs{}
reply := UptimeReply{}
err := rc.Call("Admin.Uptime", &args, &reply)
reply := ServerInfoDataReply{}
err := rc.Call(serverInfoDataRPC, &args, &reply)
if err != nil {
return time.Duration(0), err
return sid, err
}
return reply.Uptime, nil
return reply.ServerInfoData, nil
}
// GetConfig - returns config.json of the local server.
func (lc localAdminClient) GetConfig() ([]byte, error) {
if serverConfig == nil {
return nil, errors.New("config not present")
}
return json.Marshal(serverConfig)
}
// GetConfig - returns config.json of the remote server.
func (rc remoteAdminClient) GetConfig() ([]byte, error) {
args := AuthRPCArgs{}
reply := ConfigReply{}
if err := rc.Call(getConfigRPC, &args, &reply); err != nil {
return nil, err
}
return reply.Config, nil
}
// WriteTmpConfig - writes config file content to a temporary file on
// the local server.
func (lc localAdminClient) WriteTmpConfig(tmpFileName string, configBytes []byte) error {
return writeTmpConfigCommon(tmpFileName, configBytes)
}
// WriteTmpConfig - writes config file content to a temporary file on
// a remote node.
func (rc remoteAdminClient) WriteTmpConfig(tmpFileName string, configBytes []byte) error {
wArgs := WriteConfigArgs{
TmpFileName: tmpFileName,
Buf: configBytes,
}
err := rc.Call(writeTmpConfigRPC, &wArgs, &WriteConfigReply{})
if err != nil {
errorIf(err, "Failed to write temporary config file.")
return err
}
return nil
}
// CommitConfig - Move the new config in tmpFileName onto config.json
// on a local node.
func (lc localAdminClient) CommitConfig(tmpFileName string) error {
configFile := getConfigFile()
tmpConfigFile := filepath.Join(getConfigDir(), tmpFileName)
err := os.Rename(tmpConfigFile, configFile)
errorIf(err, fmt.Sprintf("Failed to rename %s to %s", tmpConfigFile, configFile))
return err
}
// CommitConfig - Move the new config in tmpFileName onto config.json
// on a remote node.
func (rc remoteAdminClient) CommitConfig(tmpFileName string) error {
cArgs := CommitConfigArgs{
FileName: tmpFileName,
}
cReply := CommitConfigReply{}
err := rc.Call(commitConfigRPC, &cArgs, &cReply)
if err != nil {
errorIf(err, "Failed to rename config file.")
return err
}
return nil
}
// adminPeer - represents an entity that implements Restart methods.
@@ -122,52 +235,43 @@ type adminPeer struct {
type adminPeers []adminPeer
// makeAdminPeers - helper function to construct a collection of adminPeer.
func makeAdminPeers(eps []*url.URL) adminPeers {
var servicePeers []adminPeer
// map to store peers that are already added to ret
seenAddr := make(map[string]bool)
// add local (self) as peer in the array
servicePeers = append(servicePeers, adminPeer{
globalMinioAddr,
func makeAdminPeers(endpoints EndpointList) (adminPeerList adminPeers) {
thisPeer := globalMinioAddr
if globalMinioHost == "" {
thisPeer = net.JoinHostPort("localhost", globalMinioPort)
}
adminPeerList = append(adminPeerList, adminPeer{
thisPeer,
localAdminClient{},
})
seenAddr[globalMinioAddr] = true
serverCred := serverConfig.GetCredential()
// iterate over endpoints to find new remote peers and add
// them to ret.
for _, ep := range eps {
if ep.Host == "" {
hostSet := set.CreateStringSet(globalMinioAddr)
cred := serverConfig.GetCredential()
serviceEndpoint := path.Join(minioReservedBucketPath, adminPath)
for _, host := range GetRemotePeers(endpoints) {
if hostSet.Contains(host) {
continue
}
// Check if the remote host has been added already
if !seenAddr[ep.Host] {
cfg := authConfig{
accessKey: serverCred.AccessKey,
secretKey: serverCred.SecretKey,
serverAddr: ep.Host,
hostSet.Add(host)
adminPeerList = append(adminPeerList, adminPeer{
addr: host,
cmdRunner: &remoteAdminClient{newAuthRPCClient(authConfig{
accessKey: cred.AccessKey,
secretKey: cred.SecretKey,
serverAddr: host,
serviceEndpoint: serviceEndpoint,
secureConn: globalIsSSL,
serviceEndpoint: path.Join(reservedBucket, adminPath),
serviceName: "Admin",
}
servicePeers = append(servicePeers, adminPeer{
addr: ep.Host,
cmdRunner: &remoteAdminClient{newAuthRPCClient(cfg)},
})
seenAddr[ep.Host] = true
}
})},
})
}
return servicePeers
return adminPeerList
}
// Initialize global adminPeer collection.
func initGlobalAdminPeers(eps []*url.URL) {
globalAdminPeers = makeAdminPeers(eps)
func initGlobalAdminPeers(endpoints EndpointList) {
globalAdminPeers = makeAdminPeers(endpoints)
}
// invokeServiceCmd - Invoke Restart command.
@@ -291,7 +395,7 @@ func getPeerUptimes(peers adminPeers) (time.Duration, error) {
// the setup is the uptime of the single minio server
// instance.
if !globalIsDistXL {
return time.Now().UTC().Sub(globalBootTime), nil
return UTCNow().Sub(globalBootTime), nil
}
uptimes := make(uptimeSlice, len(peers))
@@ -302,7 +406,8 @@ func getPeerUptimes(peers adminPeers) (time.Duration, error) {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
uptimes[idx].uptime, uptimes[idx].err = peer.cmdRunner.Uptime()
serverInfoData, rpcErr := peer.cmdRunner.ServerInfoData()
uptimes[idx].uptime, uptimes[idx].err = serverInfoData.Properties.Uptime, rpcErr
}(i, peer)
}
wg.Wait()
@@ -336,3 +441,185 @@ func getPeerUptimes(peers adminPeers) (time.Duration, error) {
return latestUptime, nil
}
// getPeerConfig - Fetches config.json from all nodes in the setup and
// returns the one that occurs in a majority of them.
func getPeerConfig(peers adminPeers) ([]byte, error) {
if !globalIsDistXL {
return peers[0].cmdRunner.GetConfig()
}
errs := make([]error, len(peers))
configs := make([][]byte, len(peers))
// Get config from all servers.
wg := sync.WaitGroup{}
for i, peer := range peers {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
configs[idx], errs[idx] = peer.cmdRunner.GetConfig()
}(i, peer)
}
wg.Wait()
// Find the maximally occurring config among peers in a
// distributed setup.
serverConfigs := make([]serverConfigV13, len(peers))
for i, configBytes := range configs {
if errs[i] != nil {
continue
}
// Unmarshal the received config files.
err := json.Unmarshal(configBytes, &serverConfigs[i])
if err != nil {
errorIf(err, "Failed to unmarshal serverConfig from ", peers[i].addr)
return nil, err
}
}
configJSON, err := getValidServerConfig(serverConfigs, errs)
if err != nil {
errorIf(err, "Unable to find a valid server config")
return nil, traceError(err)
}
// Return the config.json that was present quorum or more
// number of disks.
return json.Marshal(configJSON)
}
// getValidServerConfig - finds the server config that is present in
// quorum or more number of servers.
func getValidServerConfig(serverConfigs []serverConfigV13, errs []error) (scv serverConfigV13, e error) {
// majority-based quorum
quorum := len(serverConfigs)/2 + 1
// Count the number of disks a config.json was found in.
configCounter := make([]int, len(serverConfigs))
// We group equal serverConfigs by the lowest index of the
// same value; e.g, let us take the following serverConfigs
// in a 4-node setup,
// serverConfigs == [c1, c2, c1, c1]
// configCounter == [3, 1, 0, 0]
// c1, c2 are the only distinct values that appear. c1 is
// identified by 0, the lowest index it appears in and c2 is
// identified by 1. So, we need to find the number of times
// each of these distinct values occur.
// Invariants:
// 1. At the beginning of the i-th iteration, the number of
// unique configurations seen so far is equal to the number of
// non-zero counter values in config[:i].
// 2. At the beginning of the i-th iteration, the sum of
// elements of configCounter[:i] is equal to the number of
// non-error configurations seen so far.
// For each of the serverConfig ...
for i := range serverConfigs {
// Skip nodes where getConfig failed.
if errs[i] != nil {
continue
}
// Check if it is equal to any of the configurations
// seen so far. If j == i is reached then we have an
// unseen configuration.
for j := 0; j <= i; j++ {
if j < i && configCounter[j] == 0 {
// serverConfigs[j] is known to be
// equal to a value that was already
// seen. See example above for
// clarity.
continue
} else if j < i && reflect.DeepEqual(serverConfigs[i], serverConfigs[j]) {
// serverConfigs[i] is equal to
// serverConfigs[j], update
// serverConfigs[j]'s counter since it
// is the lower index.
configCounter[j]++
break
} else if j == i {
// serverConfigs[i] is equal to no
// other value seen before. It is
// unique so far.
configCounter[i] = 1
break
} // else invariants specified above are violated.
}
}
// We find the maximally occurring server config and check if
// there is quorum.
var configJSON serverConfigV13
maxOccurrence := 0
for i, count := range configCounter {
if maxOccurrence < count {
maxOccurrence = count
configJSON = serverConfigs[i]
}
}
// If quorum nodes don't agree.
if maxOccurrence < quorum {
return scv, errXLWriteQuorum
}
return configJSON, nil
}
// Write config contents into a temporary file on all nodes.
func writeTmpConfigPeers(peers adminPeers, tmpFileName string, configBytes []byte) []error {
// For a single-node minio server setup.
if !globalIsDistXL {
err := peers[0].cmdRunner.WriteTmpConfig(tmpFileName, configBytes)
return []error{err}
}
errs := make([]error, len(peers))
// Write config into temporary file on all nodes.
wg := sync.WaitGroup{}
for i, peer := range peers {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
errs[idx] = peer.cmdRunner.WriteTmpConfig(tmpFileName, configBytes)
}(i, peer)
}
wg.Wait()
// Return bytes written and errors (if any) during writing
// temporary config file.
return errs
}
// Move config contents from the given temporary file onto config.json
// on all nodes.
func commitConfigPeers(peers adminPeers, tmpFileName string) []error {
// For a single-node minio server setup.
if !globalIsDistXL {
return []error{peers[0].cmdRunner.CommitConfig(tmpFileName)}
}
errs := make([]error, len(peers))
// Rename temporary config file into configDir/config.json on
// all nodes.
wg := sync.WaitGroup{}
for i, peer := range peers {
wg.Add(1)
go func(idx int, peer adminPeer) {
defer wg.Done()
errs[idx] = peer.cmdRunner.CommitConfig(tmpFileName)
}(i, peer)
}
wg.Wait()
// Return errors (if any) received during rename.
return errs
}
+260
View File
@@ -0,0 +1,260 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"encoding/json"
"reflect"
"testing"
)
var (
config1 = []byte(`{
"version": "13",
"credential": {
"accessKey": "minio",
"secretKey": "minio123"
},
"region": "us-east-1",
"logger": {
"console": {
"enable": true,
"level": "debug"
},
"file": {
"enable": false,
"fileName": "",
"level": ""
}
},
"notify": {
"amqp": {
"1": {
"enable": false,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"streaming": {
"enable": false,
"clusterID": "",
"clientID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"elasticsearch": {
"1": {
"enable": false,
"url": "",
"index": ""
}
},
"redis": {
"1": {
"enable": false,
"address": "",
"password": "",
"key": ""
}
},
"postgresql": {
"1": {
"enable": false,
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": ""
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
}
}
}
}
`)
// diff from config1 - amqp.Enable is True
config2 = []byte(`{
"version": "13",
"credential": {
"accessKey": "minio",
"secretKey": "minio123"
},
"region": "us-east-1",
"logger": {
"console": {
"enable": true,
"level": "debug"
},
"file": {
"enable": false,
"fileName": "",
"level": ""
}
},
"notify": {
"amqp": {
"1": {
"enable": true,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"streaming": {
"enable": false,
"clusterID": "",
"clientID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"elasticsearch": {
"1": {
"enable": false,
"url": "",
"index": ""
}
},
"redis": {
"1": {
"enable": false,
"address": "",
"password": "",
"key": ""
}
},
"postgresql": {
"1": {
"enable": false,
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": ""
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
}
}
}
}
`)
)
// TestGetValidServerConfig - test for getValidServerConfig.
func TestGetValidServerConfig(t *testing.T) {
var c1, c2 serverConfigV13
err := json.Unmarshal(config1, &c1)
if err != nil {
t.Fatalf("json unmarshal of %s failed: %v", string(config1), err)
}
err = json.Unmarshal(config2, &c2)
if err != nil {
t.Fatalf("json unmarshal of %s failed: %v", string(config2), err)
}
// Valid config.
noErrs := []error{nil, nil, nil, nil}
serverConfigs := []serverConfigV13{c1, c2, c1, c1}
validConfig, err := getValidServerConfig(serverConfigs, noErrs)
if err != nil {
t.Errorf("Expected a valid config but received %v instead", err)
}
if !reflect.DeepEqual(validConfig, c1) {
t.Errorf("Expected valid config to be %v but received %v", config1, validConfig)
}
// Invalid config - no quorum.
serverConfigs = []serverConfigV13{c1, c2, c2, c1}
_, err = getValidServerConfig(serverConfigs, noErrs)
if err != errXLWriteQuorum {
t.Errorf("Expected to fail due to lack of quorum but received %v", err)
}
// All errors
allErrs := []error{errDiskNotFound, errDiskNotFound, errDiskNotFound, errDiskNotFound}
serverConfigs = []serverConfigV13{{}, {}, {}, {}}
_, err = getValidServerConfig(serverConfigs, allErrs)
if err != errXLWriteQuorum {
t.Errorf("Expected to fail due to lack of quorum but received %v", err)
}
}
+114 -15
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,8 +17,12 @@
package cmd
import (
"encoding/json"
"errors"
"net/rpc"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"time"
router "github.com/gorilla/mux"
@@ -48,10 +52,16 @@ type ListLocksReply struct {
volLocks []VolumeLockInfo
}
// UptimeReply - wraps the uptime response over RPC.
type UptimeReply struct {
// ServerInfoDataReply - wraps the server info response over RPC.
type ServerInfoDataReply struct {
AuthRPCReply
Uptime time.Duration
ServerInfoData ServerInfoData
}
// ConfigReply - wraps the server config response over RPC.
type ConfigReply struct {
AuthRPCReply
Config []byte // json-marshalled bytes of serverConfigV13
}
// Restart - Restart this instance of minio server.
@@ -111,8 +121,8 @@ func (s *adminCmd) ReInitDisks(args *AuthRPCArgs, reply *AuthRPCReply) error {
return nil
}
// Uptime - returns the time when object layer was initialized on this server.
func (s *adminCmd) Uptime(args *AuthRPCArgs, reply *UptimeReply) error {
// ServerInfo - returns the server info when object layer was initialized on this server.
func (s *adminCmd) ServerInfoData(args *AuthRPCArgs, reply *ServerInfoDataReply) error {
if err := args.IsAuthenticated(); err != nil {
return err
}
@@ -121,27 +131,116 @@ func (s *adminCmd) Uptime(args *AuthRPCArgs, reply *UptimeReply) error {
return errServerNotInitialized
}
// N B The uptime is computed assuming that the system time is
// monotonic. This is not the case in time pkg in Go, see
// https://github.com/golang/go/issues/12914. This is expected
// to be fixed by go1.9.
*reply = UptimeReply{
Uptime: time.Now().UTC().Sub(globalBootTime),
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
return errServerNotInitialized
}
storageInfo := objLayer.StorageInfo()
var arns []string
for queueArn := range globalEventNotifier.GetAllExternalTargets() {
arns = append(arns, queueArn)
}
reply.ServerInfoData = ServerInfoData{
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
Region: serverConfig.GetRegion(),
SQSARN: arns,
},
StorageInfo: storageInfo,
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
}
return nil
}
// GetConfig - returns the config.json of this server.
func (s *adminCmd) GetConfig(args *AuthRPCArgs, reply *ConfigReply) error {
if err := args.IsAuthenticated(); err != nil {
return err
}
if serverConfig == nil {
return errors.New("config not present")
}
jsonBytes, err := json.Marshal(serverConfig)
if err != nil {
return err
}
reply.Config = jsonBytes
return nil
}
// WriteConfigArgs - wraps the bytes to be written and temporary file name.
type WriteConfigArgs struct {
AuthRPCArgs
TmpFileName string
Buf []byte
}
// WriteConfigReply - wraps the result of a writing config into a temporary file.
// the remote node.
type WriteConfigReply struct {
AuthRPCReply
}
func writeTmpConfigCommon(tmpFileName string, configBytes []byte) error {
tmpConfigFile := filepath.Join(getConfigDir(), tmpFileName)
err := ioutil.WriteFile(tmpConfigFile, configBytes, 0666)
errorIf(err, fmt.Sprintf("Failed to write to temporary config file %s", tmpConfigFile))
return err
}
// WriteTmpConfig - writes the supplied config contents onto the
// supplied temporary file.
func (s *adminCmd) WriteTmpConfig(wArgs *WriteConfigArgs, wReply *WriteConfigReply) error {
if err := wArgs.IsAuthenticated(); err != nil {
return err
}
return writeTmpConfigCommon(wArgs.TmpFileName, wArgs.Buf)
}
// CommitConfigArgs - wraps the config file name that needs to be
// committed into config.json on this node.
type CommitConfigArgs struct {
AuthRPCArgs
FileName string
}
// CommitConfigReply - represents response to commit of config file on
// this node.
type CommitConfigReply struct {
AuthRPCReply
}
// CommitConfig - Renames the temporary file into config.json on this node.
func (s *adminCmd) CommitConfig(cArgs *CommitConfigArgs, cReply *CommitConfigReply) error {
configFile := getConfigFile()
tmpConfigFile := filepath.Join(getConfigDir(), cArgs.FileName)
err := os.Rename(tmpConfigFile, configFile)
errorIf(err, fmt.Sprintf("Failed to rename %s to %s", tmpConfigFile, configFile))
return err
}
// registerAdminRPCRouter - registers RPC methods for service status,
// stop and restart commands.
func registerAdminRPCRouter(mux *router.Router) error {
adminRPCHandler := &adminCmd{}
adminRPCServer := rpc.NewServer()
adminRPCServer := newRPCServer()
err := adminRPCServer.RegisterName("Admin", adminRPCHandler)
if err != nil {
return traceError(err)
}
adminRouter := mux.NewRoute().PathPrefix(reservedBucket).Subrouter()
adminRouter := mux.NewRoute().PathPrefix(minioReservedBucketPath).Subrouter()
adminRouter.Path(adminPath).Handler(adminRPCServer)
return nil
}
+111 -13
View File
@@ -17,9 +17,8 @@
package cmd
import (
"net/url"
"encoding/json"
"testing"
"time"
)
func testAdminCmd(cmd cmdType, t *testing.T) {
@@ -39,7 +38,7 @@ func testAdminCmd(cmd cmdType, t *testing.T) {
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: time.Now().UTC(),
RequestTime: UTCNow(),
}
reply := LoginRPCReply{}
err = adminServer.Login(&args, &reply)
@@ -52,7 +51,7 @@ func testAdminCmd(cmd cmdType, t *testing.T) {
<-globalServiceSignalCh
}()
ga := AuthRPCArgs{AuthToken: reply.AuthToken, RequestTime: time.Now().UTC()}
ga := AuthRPCArgs{AuthToken: reply.AuthToken}
genReply := AuthRPCReply{}
switch cmd {
case restartCmd:
@@ -86,9 +85,7 @@ func TestReInitDisks(t *testing.T) {
defer removeRoots(xlDirs)
// Set globalEndpoints for a single node XL setup.
for _, xlDir := range xlDirs {
globalEndpoints = append(globalEndpoints, &url.URL{Path: xlDir})
}
globalEndpoints = mustGetNewEndpointList(xlDirs...)
// Setup admin rpc server for an XL backend.
globalIsXL = true
@@ -98,7 +95,7 @@ func TestReInitDisks(t *testing.T) {
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: time.Now().UTC(),
RequestTime: UTCNow(),
}
reply := LoginRPCReply{}
err = adminServer.Login(&args, &reply)
@@ -107,8 +104,7 @@ func TestReInitDisks(t *testing.T) {
}
authArgs := AuthRPCArgs{
AuthToken: reply.AuthToken,
RequestTime: time.Now().UTC(),
AuthToken: reply.AuthToken,
}
authReply := AuthRPCReply{}
@@ -124,7 +120,7 @@ func TestReInitDisks(t *testing.T) {
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: time.Now().UTC(),
RequestTime: UTCNow(),
}
fsReply := LoginRPCReply{}
err = fsAdminServer.Login(&fsArgs, &fsReply)
@@ -133,8 +129,7 @@ func TestReInitDisks(t *testing.T) {
}
authArgs = AuthRPCArgs{
AuthToken: fsReply.AuthToken,
RequestTime: time.Now().UTC(),
AuthToken: fsReply.AuthToken,
}
authReply = AuthRPCReply{}
// Attempt ReInitDisks service on a FS backend.
@@ -144,3 +139,106 @@ func TestReInitDisks(t *testing.T) {
errUnsupportedBackend, err)
}
}
// TestGetConfig - Test for GetConfig admin RPC.
func TestGetConfig(t *testing.T) {
// Reset global variables to start afresh.
resetTestGlobals()
rootPath, err := newTestConfig("us-east-1")
if err != nil {
t.Fatalf("Unable to initialize server config. %s", err)
}
defer removeAll(rootPath)
adminServer := adminCmd{}
creds := serverConfig.GetCredential()
args := LoginRPCArgs{
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: UTCNow(),
}
reply := LoginRPCReply{}
err = adminServer.Login(&args, &reply)
if err != nil {
t.Fatalf("Failed to login to admin server - %v", err)
}
authArgs := AuthRPCArgs{
AuthToken: reply.AuthToken,
}
configReply := ConfigReply{}
err = adminServer.GetConfig(&authArgs, &configReply)
if err != nil {
t.Errorf("Expected GetConfig to pass but failed with %v", err)
}
var config serverConfigV13
err = json.Unmarshal(configReply.Config, &config)
if err != nil {
t.Errorf("Expected json unmarshal to pass but failed with %v", err)
}
}
// TestWriteAndCommitConfig - test for WriteTmpConfig and CommitConfig
// RPC handler.
func TestWriteAndCommitConfig(t *testing.T) {
// Reset global variables to start afresh.
resetTestGlobals()
rootPath, err := newTestConfig("us-east-1")
if err != nil {
t.Fatalf("Unable to initialize server config. %s", err)
}
defer removeAll(rootPath)
adminServer := adminCmd{}
creds := serverConfig.GetCredential()
args := LoginRPCArgs{
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: UTCNow(),
}
reply := LoginRPCReply{}
err = adminServer.Login(&args, &reply)
if err != nil {
t.Fatalf("Failed to login to admin server - %v", err)
}
// Write temporary config.
buf := []byte("hello")
tmpFileName := mustGetUUID()
wArgs := WriteConfigArgs{
AuthRPCArgs: AuthRPCArgs{
AuthToken: reply.AuthToken,
},
TmpFileName: tmpFileName,
Buf: buf,
}
err = adminServer.WriteTmpConfig(&wArgs, &WriteConfigReply{})
if err != nil {
t.Fatalf("Failed to write temporary config %v", err)
}
if err != nil {
t.Errorf("Expected to succeed but failed %v", err)
}
cArgs := CommitConfigArgs{
AuthRPCArgs: AuthRPCArgs{
AuthToken: reply.AuthToken,
},
FileName: tmpFileName,
}
cReply := CommitConfigReply{}
err = adminServer.CommitConfig(&cArgs, &cReply)
if err != nil {
t.Fatalf("Failed to commit config file %v", err)
}
}
+56 -3
View File
@@ -56,6 +56,8 @@ const (
ErrInvalidBucketName
ErrInvalidDigest
ErrInvalidRange
ErrInvalidCopyPartRange
ErrInvalidCopyPartRangeSource
ErrInvalidMaxKeys
ErrInvalidMaxUploads
ErrInvalidMaxParts
@@ -112,6 +114,8 @@ const (
ErrInvalidQueryParams
ErrBucketAlreadyOwnedByYou
ErrInvalidDuration
ErrNotSupported
ErrBucketAlreadyExists
// Add new error codes here.
// Bucket notification related errors.
@@ -137,6 +141,7 @@ const (
ErrObjectExistsAsDirectory
ErrPolicyNesting
ErrInvalidObjectName
ErrInvalidResourceName
ErrServerNotInitialized
// Add new extended error codes here.
// Please open a https://github.com/minio/minio/issues before adding
@@ -144,6 +149,8 @@ const (
ErrAdminInvalidAccessKey
ErrAdminInvalidSecretKey
ErrAdminConfigNoQuorum
ErrInsecureClientRequest
)
// error code to APIError structure, these fields carry respective
@@ -225,7 +232,7 @@ var errorCodeResponse = map[APIErrorCode]APIError{
HTTPStatusCode: http.StatusInternalServerError,
},
ErrInvalidAccessKeyID: {
Code: "InvalidAccessKeyID",
Code: "InvalidAccessKeyId",
Description: "The access key ID you provided does not exist in our records.",
HTTPStatusCode: http.StatusForbidden,
},
@@ -311,7 +318,7 @@ var errorCodeResponse = map[APIErrorCode]APIError{
},
ErrInvalidPart: {
Code: "InvalidPart",
Description: "One or more of the specified parts could not be found.",
Description: "One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidPartOrder: {
@@ -349,6 +356,11 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "The bucket you tried to delete is not empty",
HTTPStatusCode: http.StatusConflict,
},
ErrBucketAlreadyExists: {
Code: "BucketAlreadyExists",
Description: "The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.",
HTTPStatusCode: http.StatusConflict,
},
ErrAllAccessDisabled: {
Code: "AllAccessDisabled",
Description: "All access to this bucket has been disabled.",
@@ -539,6 +551,16 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "Configurations overlap. Configurations on the same bucket cannot share a common event type.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidCopyPartRange: {
Code: "InvalidArgument",
Description: "The x-amz-copy-source-range value must be of the form bytes=first-last where first and last are the zero-based offsets of the first and last bytes to copy",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidCopyPartRangeSource: {
Code: "InvalidArgument",
Description: "Range specified is not valid for source object",
HTTPStatusCode: http.StatusBadRequest,
},
/// S3 extensions.
ErrContentSHA256Mismatch: {
@@ -575,7 +597,12 @@ var errorCodeResponse = map[APIErrorCode]APIError{
},
ErrInvalidObjectName: {
Code: "XMinioInvalidObjectName",
Description: "Object name contains unsupported characters. Unsupported characters are `^*|\\\"",
Description: "Object name contains unsupported characters.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidResourceName: {
Code: "XMinioInvalidResourceName",
Description: "Resource name contains bad components such as \"..\" or \".\".",
HTTPStatusCode: http.StatusBadRequest,
},
ErrServerNotInitialized: {
@@ -593,6 +620,16 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "The secret key is invalid.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrAdminConfigNoQuorum: {
Code: "XMinioAdminConfigNoQuorum",
Description: "Configuration update failed because server quorum was not met",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrInsecureClientRequest: {
Code: "XMinioInsecureClientRequest",
Description: "Cannot respond to plain-text request from TLS-encrypted server",
HTTPStatusCode: http.StatusBadRequest,
},
// Add your error structure here.
}
@@ -632,6 +669,8 @@ func toAPIErrorCode(err error) (apiErr APIErrorCode) {
apiErr = ErrStorageFull
case BadDigest:
apiErr = ErrBadDigest
case AllAccessDisabled:
apiErr = ErrAllAccessDisabled
case IncompleteBody:
apiErr = ErrIncompleteBody
case ObjectExistsAsDirectory:
@@ -642,8 +681,12 @@ func toAPIErrorCode(err error) (apiErr APIErrorCode) {
apiErr = ErrInvalidBucketName
case BucketNotFound:
apiErr = ErrNoSuchBucket
case BucketAlreadyOwnedByYou:
apiErr = ErrBucketAlreadyOwnedByYou
case BucketNotEmpty:
apiErr = ErrBucketNotEmpty
case BucketAlreadyExists:
apiErr = ErrBucketAlreadyExists
case BucketExists:
apiErr = ErrBucketAlreadyOwnedByYou
case ObjectNotFound:
@@ -668,12 +711,22 @@ func toAPIErrorCode(err error) (apiErr APIErrorCode) {
apiErr = ErrNoSuchUpload
case PartTooSmall:
apiErr = ErrEntityTooSmall
case SignatureDoesNotMatch:
apiErr = ErrSignatureDoesNotMatch
case SHA256Mismatch:
apiErr = ErrContentSHA256Mismatch
case ObjectTooLarge:
apiErr = ErrEntityTooLarge
case ObjectTooSmall:
apiErr = ErrEntityTooSmall
case NotSupported:
apiErr = ErrNotSupported
case NotImplemented:
apiErr = ErrNotImplemented
case PolicyNotFound:
apiErr = ErrNoSuchBucketPolicy
case PartTooBig:
apiErr = ErrEntityTooLarge
default:
apiErr = ErrInternalError
}
+8
View File
@@ -103,6 +103,14 @@ func TestAPIErrCode(t *testing.T) {
StorageFull{},
ErrStorageFull,
},
{
NotSupported{},
ErrNotSupported,
},
{
NotImplemented{},
ErrNotImplemented,
},
{
errSignatureMismatch,
ErrSignatureDoesNotMatch,
+9 -4
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2015,2016 Minio, Inc.
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -34,8 +34,13 @@ func mustGetRequestID(t time.Time) string {
// Write http common headers
func setCommonHeaders(w http.ResponseWriter) {
// Set unique request ID for each reply.
w.Header().Set(responseRequestIDKey, mustGetRequestID(time.Now().UTC()))
w.Header().Set(responseRequestIDKey, mustGetRequestID(UTCNow()))
w.Header().Set("Server", globalServerUserAgent)
// Set `x-amz-bucket-region` only if region is set on the server
// by default minio uses an empty region.
if region := serverConfig.GetRegion(); region != "" {
w.Header().Set("X-Amz-Bucket-Region", region)
}
w.Header().Set("Accept-Ranges", "bytes")
}
@@ -61,8 +66,8 @@ func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, contentRange *h
w.Header().Set("Last-Modified", lastModified)
// Set Etag if available.
if objInfo.MD5Sum != "" {
w.Header().Set("ETag", "\""+objInfo.MD5Sum+"\"")
if objInfo.ETag != "" {
w.Header().Set("ETag", "\""+objInfo.ETag+"\"")
}
// Set all other user defined metadata.
+2 -3
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -18,12 +18,11 @@ package cmd
import (
"testing"
"time"
)
func TestNewRequestID(t *testing.T) {
// Ensure that it returns an alphanumeric result of length 16.
var id = mustGetRequestID(time.Now().UTC())
var id = mustGetRequestID(UTCNow())
if len(id) != 16 {
t.Fail()
+20 -25
View File
@@ -166,12 +166,13 @@ type ListBucketsResponse struct {
// Upload container for in progress multipart upload
type Upload struct {
Key string
UploadID string `xml:"UploadId"`
Initiator Initiator
Owner Owner
StorageClass string
Initiated string
Key string
UploadID string `xml:"UploadId"`
Initiator Initiator
Owner Owner
StorageClass string
Initiated string
HealUploadInfo *HealObjectInfo `xml:"HealObjectInfo,omitempty"`
}
// CommonPrefix container for prefix response in ListObjectsResponse
@@ -287,8 +288,6 @@ func generateListBucketsResponse(buckets []BucketInfo) ListBucketsResponse {
var owner = Owner{}
owner.ID = globalMinioDefaultOwnerID
owner.DisplayName = globalMinioDefaultOwnerID
for _, bucket := range buckets {
var listbucket = Bucket{}
listbucket.Name = bucket.Name
@@ -311,8 +310,6 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
var data = ListObjectsResponse{}
owner.ID = globalMinioDefaultOwnerID
owner.DisplayName = globalMinioDefaultOwnerID
for _, object := range resp.Objects {
var content = Object{}
if object.Name == "" {
@@ -320,8 +317,8 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
}
content.Key = object.Name
content.LastModified = object.ModTime.UTC().Format(timeFormatAMZLong)
if object.MD5Sum != "" {
content.ETag = "\"" + object.MD5Sum + "\""
if object.ETag != "" {
content.ETag = "\"" + object.ETag + "\""
}
content.Size = object.Size
content.StorageClass = globalMinioDefaultStorageClass
@@ -351,26 +348,25 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
}
// generates an ListObjectsV2 response for the said bucket with other enumerated options.
func generateListObjectsV2Response(bucket, prefix, token, startAfter, delimiter string, fetchOwner bool, maxKeys int, resp ListObjectsInfo) ListObjectsV2Response {
func generateListObjectsV2Response(bucket, prefix, token, nextToken, startAfter, delimiter string, fetchOwner, isTruncated bool, maxKeys int, objects []ObjectInfo, prefixes []string) ListObjectsV2Response {
var contents []Object
var prefixes []CommonPrefix
var commonPrefixes []CommonPrefix
var owner = Owner{}
var data = ListObjectsV2Response{}
if fetchOwner {
owner.ID = globalMinioDefaultOwnerID
owner.DisplayName = globalMinioDefaultOwnerID
}
for _, object := range resp.Objects {
for _, object := range objects {
var content = Object{}
if object.Name == "" {
continue
}
content.Key = object.Name
content.LastModified = object.ModTime.UTC().Format(timeFormatAMZLong)
if object.MD5Sum != "" {
content.ETag = "\"" + object.MD5Sum + "\""
if object.ETag != "" {
content.ETag = "\"" + object.ETag + "\""
}
content.Size = object.Size
content.StorageClass = globalMinioDefaultStorageClass
@@ -386,14 +382,14 @@ func generateListObjectsV2Response(bucket, prefix, token, startAfter, delimiter
data.Prefix = prefix
data.MaxKeys = maxKeys
data.ContinuationToken = token
data.NextContinuationToken = resp.NextMarker
data.IsTruncated = resp.IsTruncated
for _, prefix := range resp.Prefixes {
data.NextContinuationToken = nextToken
data.IsTruncated = isTruncated
for _, prefix := range prefixes {
var prefixItem = CommonPrefix{}
prefixItem.Prefix = prefix
prefixes = append(prefixes, prefixItem)
commonPrefixes = append(commonPrefixes, prefixItem)
}
data.CommonPrefixes = prefixes
data.CommonPrefixes = commonPrefixes
data.KeyCount = len(data.Contents) + len(data.CommonPrefixes)
return data
}
@@ -442,9 +438,7 @@ func generateListPartsResponse(partsInfo ListPartsInfo) ListPartsResponse {
listPartsResponse.UploadID = partsInfo.UploadID
listPartsResponse.StorageClass = globalMinioDefaultStorageClass
listPartsResponse.Initiator.ID = globalMinioDefaultOwnerID
listPartsResponse.Initiator.DisplayName = globalMinioDefaultOwnerID
listPartsResponse.Owner.ID = globalMinioDefaultOwnerID
listPartsResponse.Owner.DisplayName = globalMinioDefaultOwnerID
listPartsResponse.MaxParts = partsInfo.MaxParts
listPartsResponse.PartNumberMarker = partsInfo.PartNumberMarker
@@ -488,6 +482,7 @@ func generateListMultipartUploadsResponse(bucket string, multipartsInfo ListMult
newUpload.UploadID = upload.UploadID
newUpload.Key = upload.Object
newUpload.Initiated = upload.Initiated.UTC().Format(timeFormatAMZLong)
newUpload.HealUploadInfo = upload.HealUploadInfo
listMultipartUploadsResponse.Uploads[index] = newUpload
}
return listMultipartUploadsResponse
+32 -27
View File
@@ -63,7 +63,8 @@ func isRequestPostPolicySignatureV4(r *http.Request) bool {
// Verify if the request has AWS Streaming Signature Version '4'. This is only valid for 'PUT' operation.
func isRequestSignStreamingV4(r *http.Request) bool {
return r.Header.Get("x-amz-content-sha256") == streamingContentSHA256 && r.Method == httpPUT
return r.Header.Get("x-amz-content-sha256") == streamingContentSHA256 &&
r.Method == httpPUT
}
// Authorization type.
@@ -112,13 +113,13 @@ func checkRequestAuthType(r *http.Request, bucket, policyAction, region string)
// Signature V2 validation.
s3Error := isReqAuthenticatedV2(r)
if s3Error != ErrNone {
errorIf(errSignatureMismatch, dumpRequest(r))
errorIf(errSignatureMismatch, "%s", dumpRequest(r))
}
return s3Error
case authTypeSigned, authTypePresigned:
s3Error := isReqAuthenticated(r, region)
if s3Error != ErrNone {
errorIf(errSignatureMismatch, dumpRequest(r))
errorIf(errSignatureMismatch, "%s", dumpRequest(r))
}
return s3Error
}
@@ -141,19 +142,16 @@ func isReqAuthenticatedV2(r *http.Request) (s3Error APIErrorCode) {
return doesPresignV2SignatureMatch(r)
}
func reqSignatureV4Verify(r *http.Request) (s3Error APIErrorCode) {
sha256sum := r.Header.Get("X-Amz-Content-Sha256")
// Skips calculating sha256 on the payload on server,
// if client requested for it.
if skipContentSha256Cksum(r) {
sha256sum = unsignedPayload
func reqSignatureV4Verify(r *http.Request, region string) (s3Error APIErrorCode) {
sha256sum := getContentSha256Cksum(r)
switch {
case isRequestSignatureV4(r):
return doesSignatureMatch(sha256sum, r, region)
case isRequestPresignedSignatureV4(r):
return doesPresignedSignatureMatch(sha256sum, r, region)
default:
return ErrAccessDenied
}
if isRequestSignatureV4(r) {
return doesSignatureMatch(sha256sum, r, serverConfig.GetRegion())
} else if isRequestPresignedSignatureV4(r) {
return doesPresignedSignatureMatch(sha256sum, r, serverConfig.GetRegion())
}
return ErrAccessDenied
}
// Verify if request has valid AWS Signature Version '4'.
@@ -161,32 +159,39 @@ func isReqAuthenticated(r *http.Request, region string) (s3Error APIErrorCode) {
if r == nil {
return ErrInternalError
}
if errCode := reqSignatureV4Verify(r, region); errCode != ErrNone {
return errCode
}
payload, err := ioutil.ReadAll(r.Body)
if err != nil {
errorIf(err, "Unable to read request body for signature verification")
return ErrInternalError
}
// Populate back the payload.
r.Body = ioutil.NopCloser(bytes.NewReader(payload))
// Verify Content-Md5, if payload is set.
if r.Header.Get("Content-Md5") != "" {
if r.Header.Get("Content-Md5") != getMD5HashBase64(payload) {
return ErrBadDigest
}
}
// Populate back the payload.
r.Body = ioutil.NopCloser(bytes.NewReader(payload))
// Skips calculating sha256 on the payload on server, if client requested for it.
var sha256sum string
if skipContentSha256Cksum(r) {
sha256sum = unsignedPayload
} else {
sha256sum = getSHA256Hash(payload)
return ErrNone
}
if isRequestSignatureV4(r) {
return doesSignatureMatch(sha256sum, r, region)
} else if isRequestPresignedSignatureV4(r) {
return doesPresignedSignatureMatch(sha256sum, r, region)
// Verify that X-Amz-Content-Sha256 Header == sha256(payload)
// If X-Amz-Content-Sha256 header is not sent then we don't calculate/verify sha256(payload)
sum := r.Header.Get("X-Amz-Content-Sha256")
if isRequestPresignedSignatureV4(r) {
sum = r.URL.Query().Get("X-Amz-Content-Sha256")
}
return ErrAccessDenied
if sum != "" && sum != getSHA256Hash(payload) {
return ErrContentSHA256Mismatch
}
return ErrNone
}
// authHandler - handles all the incoming authorization headers and validates them if possible.
+17 -5
View File
@@ -43,6 +43,7 @@ func TestGetRequestAuthType(t *testing.T) {
Header: http.Header{
"Authorization": []string{"AWS4-HMAC-SHA256 <cred_string>"},
"X-Amz-Content-Sha256": []string{streamingContentSHA256},
"Content-Encoding": []string{streamingContentEncoding},
},
Method: "PUT",
},
@@ -307,6 +308,16 @@ func mustNewSignedRequest(method string, urlStr string, contentLength int64, bod
return req
}
func mustNewSignedBadMD5Request(method string, urlStr string, contentLength int64, body io.ReadSeeker, t *testing.T) *http.Request {
req := mustNewRequest(method, urlStr, contentLength, body, t)
req.Header.Set("Content-Md5", "YWFhYWFhYWFhYWFhYWFhCg==")
cred := serverConfig.GetCredential()
if err := signRequestV4(req, cred.AccessKey, cred.SecretKey); err != nil {
t.Fatalf("Unable to initialized new signed http request %s", err)
}
return req
}
// Tests is requested authenticated function, tests replies for s3 errors.
func TestIsReqAuthenticated(t *testing.T) {
path, err := newTestConfig(globalMinioDefaultRegion)
@@ -315,7 +326,11 @@ func TestIsReqAuthenticated(t *testing.T) {
}
defer removeAll(path)
creds := newCredentialWithKeys("myuser", "mypassword")
creds, err := createCredential("myuser", "mypassword")
if err != nil {
t.Fatalf("unable create credential, %s", err)
}
serverConfig.SetCredential(creds)
// List of test cases for validating http request authentication.
@@ -328,16 +343,13 @@ func TestIsReqAuthenticated(t *testing.T) {
// When request is unsigned, access denied is returned.
{mustNewRequest("GET", "http://127.0.0.1:9000", 0, nil, t), ErrAccessDenied},
// When request is properly signed, but has bad Content-MD5 header.
{mustNewSignedRequest("PUT", "http://127.0.0.1:9000", 5, bytes.NewReader([]byte("hello")), t), ErrBadDigest},
{mustNewSignedBadMD5Request("PUT", "http://127.0.0.1:9000/", 5, bytes.NewReader([]byte("hello")), t), ErrBadDigest},
// When request is properly signed, error is none.
{mustNewSignedRequest("GET", "http://127.0.0.1:9000", 0, nil, t), ErrNone},
}
// Validates all testcases.
for _, testCase := range testCases {
if testCase.s3Error == ErrBadDigest {
testCase.req.Header.Set("Content-Md5", "garbage")
}
if s3Error := isReqAuthenticated(testCase.req, serverConfig.GetRegion()); s3Error != testCase.s3Error {
t.Fatalf("Unexpected s3error returned wanted %d, got %d", testCase.s3Error, s3Error)
}
+47 -38
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -52,10 +52,10 @@ type authConfig struct {
// AuthRPCClient is a authenticated RPC client which does authentication before doing Call().
type AuthRPCClient struct {
sync.Mutex // Mutex to lock this object.
rpcClient *RPCClient // Reconnectable RPC client to make any RPC call.
config authConfig // Authentication configuration information.
authToken string // Authentication token.
sync.RWMutex // Mutex to lock this object.
rpcClient *RPCClient // Reconnectable RPC client to make any RPC call.
config authConfig // Authentication configuration information.
authToken string // Authentication token.
}
// newAuthRPCClient - returns a JWT based authenticated (go) rpc client, which does automatic reconnect.
@@ -78,57 +78,66 @@ func newAuthRPCClient(config authConfig) *AuthRPCClient {
}
}
// Login - a jwt based authentication is performed with rpc server.
// Login a JWT based authentication is performed with rpc server.
func (authClient *AuthRPCClient) Login() (err error) {
// Login should be attempted one at a time.
//
// The reason for large region lock here is
// to avoid two simultaneous login attempts
// racing over each other.
//
// #1 Login() gets the lock proceeds to login.
// #2 Login() waits for the unlock to happen
// after login in #1.
// #1 Successfully completes login saves the
// newly acquired token.
// #2 Successfully gets the lock and proceeds,
// but since we have acquired the token
// already the call quickly returns.
authClient.Lock()
defer authClient.Unlock()
// Return if already logged in.
if authClient.authToken != "" {
return nil
// Attempt to login if not logged in already.
if authClient.authToken == "" {
// Login to authenticate and acquire a new auth token.
var (
loginMethod = authClient.config.serviceName + loginMethodName
loginArgs = LoginRPCArgs{
Username: authClient.config.accessKey,
Password: authClient.config.secretKey,
Version: Version,
RequestTime: UTCNow(),
}
loginReply = LoginRPCReply{}
)
if err = authClient.rpcClient.Call(loginMethod, &loginArgs, &loginReply); err != nil {
return err
}
authClient.authToken = loginReply.AuthToken
}
// Call login.
args := LoginRPCArgs{
Username: authClient.config.accessKey,
Password: authClient.config.secretKey,
Version: Version,
RequestTime: time.Now().UTC(),
}
reply := LoginRPCReply{}
serviceMethod := authClient.config.serviceName + loginMethodName
if err = authClient.rpcClient.Call(serviceMethod, &args, &reply); err != nil {
return err
}
// Logged in successfully.
authClient.authToken = reply.AuthToken
return nil
}
// call makes a RPC call after logs into the server.
func (authClient *AuthRPCClient) call(serviceMethod string, args interface {
SetAuthToken(authToken string)
SetRequestTime(requestTime time.Time)
}, reply interface{}) (err error) {
// On successful login, execute RPC call.
if err = authClient.Login(); err == nil {
// Set token and timestamp before the rpc call.
args.SetAuthToken(authClient.authToken)
args.SetRequestTime(time.Now().UTC())
if err = authClient.Login(); err != nil {
return err
} // On successful login, execute RPC call.
// Do RPC call.
err = authClient.rpcClient.Call(serviceMethod, args, reply)
}
return err
authClient.RLock()
// Set token before the rpc call.
args.SetAuthToken(authClient.authToken)
authClient.RUnlock()
// Do an RPC call.
return authClient.rpcClient.Call(serviceMethod, args, reply)
}
// Call executes RPC call till success or globalAuthRPCRetryThreshold on ErrShutdown.
func (authClient *AuthRPCClient) Call(serviceMethod string, args interface {
SetAuthToken(authToken string)
SetRequestTime(requestTime time.Time)
}, reply interface{}) (err error) {
// Done channel is used to close any lingering retry routine, as soon
+1 -1
View File
@@ -77,7 +77,7 @@ func TestLogin(t *testing.T) {
// Invalid password length
{
args: LoginRPCArgs{
Username: globalMinioDefaultOwnerID,
Username: "minio",
Password: "aaa",
Version: Version,
},
+30 -47
View File
@@ -23,30 +23,13 @@ import (
"math/rand"
"strconv"
"testing"
"time"
humanize "github.com/dustin/go-humanize"
)
// Prepare benchmark backend
// Prepare XL/FS backend for benchmark.
func prepareBenchmarkBackend(instanceType string) (ObjectLayer, []string, error) {
switch instanceType {
// Total number of disks for FS backend is set to 1.
case FSTestStr:
obj, disk, err := prepareFS()
if err != nil {
return nil, nil, err
}
return obj, []string{disk}, nil
// Total number of disks for XL backend is set to 16.
case XLTestStr:
return prepareXL()
}
obj, disk, err := prepareFS()
if err != nil {
return nil, nil, err
}
return obj, []string{disk}, nil
return prepareTestBackend(instanceType)
}
// Benchmark utility functions for ObjectLayer.PutObject().
@@ -56,7 +39,7 @@ func runPutObjectBenchmark(b *testing.B, obj ObjectLayer, objSize int) {
// obtains random bucket name.
bucket := getRandomBucketName()
// create bucket.
err = obj.MakeBucket(bucket)
err = obj.MakeBucketWithLocation(bucket, "")
if err != nil {
b.Fatal(err)
}
@@ -66,7 +49,7 @@ func runPutObjectBenchmark(b *testing.B, obj ObjectLayer, objSize int) {
// generate md5sum for the generated data.
// md5sum of the data to written is required as input for PutObject.
metadata := make(map[string]string)
metadata["md5Sum"] = getMD5Hash(textData)
metadata["etag"] = getMD5Hash(textData)
sha256sum := ""
// benchmark utility which helps obtain number of allocations and bytes allocated per ops.
b.ReportAllocs()
@@ -78,8 +61,8 @@ func runPutObjectBenchmark(b *testing.B, obj ObjectLayer, objSize int) {
if err != nil {
b.Fatal(err)
}
if objInfo.MD5Sum != metadata["md5Sum"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, objInfo.MD5Sum, metadata["md5Sum"])
if objInfo.ETag != metadata["etag"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, objInfo.ETag, metadata["etag"])
}
}
// Benchmark ends here. Stop timer.
@@ -95,22 +78,22 @@ func runPutObjectPartBenchmark(b *testing.B, obj ObjectLayer, partSize int) {
object := getRandomObjectName()
// create bucket.
err = obj.MakeBucket(bucket)
err = obj.MakeBucketWithLocation(bucket, "")
if err != nil {
b.Fatal(err)
}
objSize := 128 * humanize.MiByte
// PutObjectPart returns md5Sum of the object inserted.
// md5Sum variable is assigned with that value.
var md5Sum, uploadID string
// PutObjectPart returns etag of the object inserted.
// etag variable is assigned with that value.
var etag, uploadID string
// get text data generated for number of bytes equal to object size.
textData := generateBytesData(objSize)
// generate md5sum for the generated data.
// md5sum of the data to written is required as input for NewMultipartUpload.
metadata := make(map[string]string)
metadata["md5Sum"] = getMD5Hash(textData)
metadata["etag"] = getMD5Hash(textData)
sha256sum := ""
uploadID, err = obj.NewMultipartUpload(bucket, object, metadata)
if err != nil {
@@ -132,14 +115,14 @@ func runPutObjectPartBenchmark(b *testing.B, obj ObjectLayer, partSize int) {
textPartData = textData[j*partSize:]
}
metadata := make(map[string]string)
metadata["md5Sum"] = getMD5Hash([]byte(textPartData))
metadata["etag"] = getMD5Hash([]byte(textPartData))
var partInfo PartInfo
partInfo, err = obj.PutObjectPart(bucket, object, uploadID, j, int64(len(textPartData)), bytes.NewBuffer(textPartData), metadata["md5Sum"], sha256sum)
partInfo, err = obj.PutObjectPart(bucket, object, uploadID, j, int64(len(textPartData)), bytes.NewBuffer(textPartData), metadata["etag"], sha256sum)
if err != nil {
b.Fatal(err)
}
if partInfo.ETag != metadata["md5Sum"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, md5Sum, metadata["md5Sum"])
if partInfo.ETag != metadata["etag"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, etag, metadata["etag"])
}
}
}
@@ -216,7 +199,7 @@ func runGetObjectBenchmark(b *testing.B, obj ObjectLayer, objSize int) {
// obtains random bucket name.
bucket := getRandomBucketName()
// create bucket.
err = obj.MakeBucket(bucket)
err = obj.MakeBucketWithLocation(bucket, "")
if err != nil {
b.Fatal(err)
}
@@ -225,19 +208,19 @@ func runGetObjectBenchmark(b *testing.B, obj ObjectLayer, objSize int) {
for i := 0; i < 10; i++ {
// get text data generated for number of bytes equal to object size.
textData := generateBytesData(objSize)
// generate md5sum for the generated data.
// md5sum of the data to written is required as input for PutObject.
// generate etag for the generated data.
// etag of the data to written is required as input for PutObject.
// PutObject is the functions which writes the data onto the FS/XL backend.
metadata := make(map[string]string)
metadata["md5Sum"] = getMD5Hash(textData)
metadata["etag"] = getMD5Hash(textData)
// insert the object.
var objInfo ObjectInfo
objInfo, err = obj.PutObject(bucket, "object"+strconv.Itoa(i), int64(len(textData)), bytes.NewBuffer(textData), metadata, sha256sum)
if err != nil {
b.Fatal(err)
}
if objInfo.MD5Sum != metadata["md5Sum"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, objInfo.MD5Sum, metadata["md5Sum"])
if objInfo.ETag != metadata["etag"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, objInfo.ETag, metadata["etag"])
}
}
@@ -261,7 +244,7 @@ func runGetObjectBenchmark(b *testing.B, obj ObjectLayer, objSize int) {
func getRandomByte() []byte {
const letterBytes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
// seeding the random number generator.
rand.Seed(time.Now().UTC().UnixNano())
rand.Seed(UTCNow().UnixNano())
var b byte
// pick a character randomly.
b = letterBytes[rand.Intn(len(letterBytes))]
@@ -324,7 +307,7 @@ func runPutObjectBenchmarkParallel(b *testing.B, obj ObjectLayer, objSize int) {
// obtains random bucket name.
bucket := getRandomBucketName()
// create bucket.
err = obj.MakeBucket(bucket)
err = obj.MakeBucketWithLocation(bucket, "")
if err != nil {
b.Fatal(err)
}
@@ -334,7 +317,7 @@ func runPutObjectBenchmarkParallel(b *testing.B, obj ObjectLayer, objSize int) {
// generate md5sum for the generated data.
// md5sum of the data to written is required as input for PutObject.
metadata := make(map[string]string)
metadata["md5Sum"] = getMD5Hash([]byte(textData))
metadata["etag"] = getMD5Hash([]byte(textData))
sha256sum := ""
// benchmark utility which helps obtain number of allocations and bytes allocated per ops.
b.ReportAllocs()
@@ -349,8 +332,8 @@ func runPutObjectBenchmarkParallel(b *testing.B, obj ObjectLayer, objSize int) {
if err != nil {
b.Fatal(err)
}
if objInfo.MD5Sum != metadata["md5Sum"] {
b.Fatalf("Write no: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", objInfo.MD5Sum, metadata["md5Sum"])
if objInfo.ETag != metadata["etag"] {
b.Fatalf("Write no: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", objInfo.ETag, metadata["etag"])
}
i++
}
@@ -372,7 +355,7 @@ func runGetObjectBenchmarkParallel(b *testing.B, obj ObjectLayer, objSize int) {
// obtains random bucket name.
bucket := getRandomBucketName()
// create bucket.
err = obj.MakeBucket(bucket)
err = obj.MakeBucketWithLocation(bucket, "")
if err != nil {
b.Fatal(err)
}
@@ -384,7 +367,7 @@ func runGetObjectBenchmarkParallel(b *testing.B, obj ObjectLayer, objSize int) {
// md5sum of the data to written is required as input for PutObject.
// PutObject is the functions which writes the data onto the FS/XL backend.
metadata := make(map[string]string)
metadata["md5Sum"] = getMD5Hash([]byte(textData))
metadata["etag"] = getMD5Hash([]byte(textData))
sha256sum := ""
// insert the object.
var objInfo ObjectInfo
@@ -392,8 +375,8 @@ func runGetObjectBenchmarkParallel(b *testing.B, obj ObjectLayer, objSize int) {
if err != nil {
b.Fatal(err)
}
if objInfo.MD5Sum != metadata["md5Sum"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, objInfo.MD5Sum, metadata["md5Sum"])
if objInfo.ETag != metadata["etag"] {
b.Fatalf("Write no: %d: Md5Sum mismatch during object write into the bucket: Expected %s, got %s", i+1, objInfo.ETag, metadata["etag"])
}
}
+68
View File
@@ -0,0 +1,68 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"encoding/json"
"fmt"
)
// BrowserFlag - wrapper bool type.
type BrowserFlag bool
// String - returns string of BrowserFlag.
func (bf BrowserFlag) String() string {
if bf {
return "on"
}
return "off"
}
// MarshalJSON - converts BrowserFlag into JSON data.
func (bf BrowserFlag) MarshalJSON() ([]byte, error) {
return json.Marshal(bf.String())
}
// UnmarshalJSON - parses given data into BrowserFlag.
func (bf *BrowserFlag) UnmarshalJSON(data []byte) (err error) {
var s string
if err = json.Unmarshal(data, &s); err == nil {
b := BrowserFlag(true)
if s == "" {
// Empty string is treated as valid.
*bf = b
} else if b, err = ParseBrowserFlag(s); err == nil {
*bf = b
}
}
return err
}
// ParseBrowserFlag - parses string into BrowserFlag.
func ParseBrowserFlag(s string) (bf BrowserFlag, err error) {
if s == "on" {
bf = true
} else if s == "off" {
bf = false
} else {
err = fmt.Errorf("invalid value %s for BrowserFlag", s)
}
return bf, err
}
+137
View File
@@ -0,0 +1,137 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"errors"
"testing"
)
// Test BrowserFlag.String()
func TestBrowserFlagString(t *testing.T) {
var bf BrowserFlag
testCases := []struct {
flag BrowserFlag
expectedResult string
}{
{bf, "off"},
{BrowserFlag(true), "on"},
{BrowserFlag(false), "off"},
}
for _, testCase := range testCases {
str := testCase.flag.String()
if testCase.expectedResult != str {
t.Fatalf("expected: %v, got: %v", testCase.expectedResult, str)
}
}
}
// Test BrowserFlag.MarshalJSON()
func TestBrowserFlagMarshalJSON(t *testing.T) {
var bf BrowserFlag
testCases := []struct {
flag BrowserFlag
expectedResult string
}{
{bf, `"off"`},
{BrowserFlag(true), `"on"`},
{BrowserFlag(false), `"off"`},
}
for _, testCase := range testCases {
data, _ := testCase.flag.MarshalJSON()
if testCase.expectedResult != string(data) {
t.Fatalf("expected: %v, got: %v", testCase.expectedResult, string(data))
}
}
}
// Test BrowserFlag.UnmarshalJSON()
func TestBrowserFlagUnmarshalJSON(t *testing.T) {
testCases := []struct {
data []byte
expectedResult BrowserFlag
expectedErr error
}{
{[]byte(`{}`), BrowserFlag(false), errors.New("json: cannot unmarshal object into Go value of type string")},
{[]byte(`["on"]`), BrowserFlag(false), errors.New("json: cannot unmarshal array into Go value of type string")},
{[]byte(`"junk"`), BrowserFlag(false), errors.New("invalid value junk for BrowserFlag")},
{[]byte(`"true"`), BrowserFlag(false), errors.New("invalid value true for BrowserFlag")},
{[]byte(`"false"`), BrowserFlag(false), errors.New("invalid value false for BrowserFlag")},
{[]byte(`"ON"`), BrowserFlag(false), errors.New("invalid value ON for BrowserFlag")},
{[]byte(`"OFF"`), BrowserFlag(false), errors.New("invalid value OFF for BrowserFlag")},
{[]byte(`""`), BrowserFlag(true), nil},
{[]byte(`"on"`), BrowserFlag(true), nil},
{[]byte(`"off"`), BrowserFlag(false), nil},
}
for _, testCase := range testCases {
var flag BrowserFlag
err := (&flag).UnmarshalJSON(testCase.data)
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
} else if err == nil {
t.Fatalf("error: expected = %v, got = <nil>", testCase.expectedErr)
} else if testCase.expectedErr.Error() != err.Error() {
t.Fatalf("error: expected = %v, got = %v", testCase.expectedErr, err)
}
if err == nil && testCase.expectedResult != flag {
t.Fatalf("result: expected: %v, got: %v", testCase.expectedResult, flag)
}
}
}
// Test ParseBrowserFlag()
func TestParseBrowserFlag(t *testing.T) {
testCases := []struct {
flagStr string
expectedResult BrowserFlag
expectedErr error
}{
{"", BrowserFlag(false), errors.New("invalid value ‘’ for BrowserFlag")},
{"junk", BrowserFlag(false), errors.New("invalid value junk for BrowserFlag")},
{"true", BrowserFlag(false), errors.New("invalid value true for BrowserFlag")},
{"false", BrowserFlag(false), errors.New("invalid value false for BrowserFlag")},
{"ON", BrowserFlag(false), errors.New("invalid value ON for BrowserFlag")},
{"OFF", BrowserFlag(false), errors.New("invalid value OFF for BrowserFlag")},
{"on", BrowserFlag(true), nil},
{"off", BrowserFlag(false), nil},
}
for _, testCase := range testCases {
bf, err := ParseBrowserFlag(testCase.flagStr)
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
} else if err == nil {
t.Fatalf("error: expected = %v, got = <nil>", testCase.expectedErr)
} else if testCase.expectedErr.Error() != err.Error() {
t.Fatalf("error: expected = %v, got = %v", testCase.expectedErr, err)
}
if err == nil && testCase.expectedResult != bf {
t.Fatalf("result: expected: %v, got: %v", testCase.expectedResult, bf)
}
}
}
+5 -4
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,6 +17,7 @@
package cmd
import (
"fmt"
"path"
"sync"
"time"
@@ -63,8 +64,8 @@ func (br *browserPeerAPIHandlers) SetAuthPeer(args SetAuthPeerArgs, reply *AuthR
return err
}
if err := validateAuthKeys(args.Creds.AccessKey, args.Creds.SecretKey); err != nil {
return err
if !args.Creds.IsValid() {
return fmt.Errorf("Invalid credential passed")
}
// Update credentials in memory
@@ -111,7 +112,7 @@ func updateCredsOnPeers(creds credential) map[string]error {
secretKey: serverCred.SecretKey,
serverAddr: peers[ix],
secureConn: globalIsSSL,
serviceEndpoint: path.Join(reservedBucket, browserPeerPath),
serviceEndpoint: path.Join(minioReservedBucketPath, browserPeerPath),
serviceName: "BrowserPeer",
})
+8 -9
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -19,7 +19,6 @@ package cmd
import (
"path"
"testing"
"time"
)
// API suite container common to both FS and XL.
@@ -36,7 +35,7 @@ func (s *TestRPCBrowserPeerSuite) SetUpSuite(c *testing.T) {
serverAddr: s.testServer.Server.Listener.Addr().String(),
accessKey: s.testServer.AccessKey,
secretKey: s.testServer.SecretKey,
serviceEndpoint: path.Join(reservedBucket, browserPeerPath),
serviceEndpoint: path.Join(minioReservedBucketPath, browserPeerPath),
serviceName: "BrowserPeer",
}
}
@@ -63,9 +62,9 @@ func TestBrowserPeerRPC(t *testing.T) {
// Tests for browser peer rpc.
func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
// Construct RPC call arguments.
creds := credential{
AccessKey: "abcd1",
SecretKey: "abcd1234",
creds, err := createCredential("abcd1", "abcd1234")
if err != nil {
t.Fatalf("unable to create credential. %v", err)
}
// Validate for invalid token.
@@ -73,7 +72,7 @@ func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
args.AuthToken = "garbage"
rclient := newRPCClient(s.testAuthConf.serverAddr, s.testAuthConf.serviceEndpoint, false)
defer rclient.Close()
err := rclient.Call("BrowserPeer.SetAuthPeer", &args, &AuthRPCReply{})
err = rclient.Call("BrowserPeer.SetAuthPeer", &args, &AuthRPCReply{})
if err != nil {
if err.Error() != errInvalidToken.Error() {
t.Fatal(err)
@@ -96,7 +95,7 @@ func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: time.Now().UTC(),
RequestTime: UTCNow(),
}
rreply := &LoginRPCReply{}
err = rclient.Call("BrowserPeer"+loginMethodName, rargs, rreply)
@@ -111,7 +110,7 @@ func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
Username: creds.AccessKey,
Password: creds.SecretKey,
Version: Version,
RequestTime: time.Now().UTC(),
RequestTime: UTCNow(),
}
rreply = &LoginRPCReply{}
err = rclient.Call("BrowserPeer"+loginMethodName, rargs, rreply)
+2 -4
View File
@@ -17,8 +17,6 @@
package cmd
import (
"net/rpc"
router "github.com/gorilla/mux"
)
@@ -39,13 +37,13 @@ type browserPeerAPIHandlers struct {
func registerBrowserPeerRPCRouter(mux *router.Router) error {
bpHandlers := &browserPeerAPIHandlers{}
bpRPCServer := rpc.NewServer()
bpRPCServer := newRPCServer()
err := bpRPCServer.RegisterName("BrowserPeer", bpHandlers)
if err != nil {
return traceError(err)
}
bpRouter := mux.NewRoute().PathPrefix(reservedBucket).Subrouter()
bpRouter := mux.NewRoute().PathPrefix(minioReservedBucketPath).Subrouter()
bpRouter.Path(browserPeerPath).Handler(bpRPCServer)
return nil
}
+1 -1
View File
@@ -100,7 +100,7 @@ func (api objectAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *http
return
}
response := generateListObjectsV2Response(bucket, prefix, token, startAfter, delimiter, fetchOwner, maxKeys, listObjectsInfo)
response := generateListObjectsV2Response(bucket, prefix, token, listObjectsInfo.NextMarker, startAfter, delimiter, fetchOwner, listObjectsInfo.IsTruncated, maxKeys, listObjectsInfo.Objects, listObjectsInfo.Prefixes)
// Write success response.
writeSuccessResponseXML(w, encodeResponse(response))
+105 -58
View File
@@ -19,8 +19,8 @@ package cmd
import (
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"net"
"net/http"
"net/url"
"path"
@@ -50,6 +50,10 @@ func enforceBucketPolicy(bucket, action, resource, referer string, queryParams u
return ErrInternalError
}
if globalBucketPolicies == nil {
return ErrAccessDenied
}
// Fetch bucket policy, if policy is not set return access denied.
policy := globalBucketPolicies.GetBucketPolicy(bucket)
if policy == nil {
@@ -79,6 +83,10 @@ func enforceBucketPolicy(bucket, action, resource, referer string, queryParams u
// Check if the action is allowed on the bucket/prefix.
func isBucketActionAllowed(action, bucket, prefix string) bool {
if globalBucketPolicies == nil {
return false
}
policy := globalBucketPolicies.GetBucketPolicy(bucket)
if policy == nil {
return false
@@ -86,10 +94,7 @@ func isBucketActionAllowed(action, bucket, prefix string) bool {
resource := bucketARNPrefix + path.Join(bucket, prefix)
var conditionKeyMap map[string]set.StringSet
// Validate action, resource and conditions with current policy statements.
if !bucketPolicyEvalStatements(action, resource, conditionKeyMap, policy.Statements) {
return false
}
return true
return bucketPolicyEvalStatements(action, resource, conditionKeyMap, policy.Statements)
}
// GetBucketLocationHandler - GET Bucket location.
@@ -277,7 +282,11 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
for index, object := range deleteObjects.Objects {
wg.Add(1)
go func(i int, obj ObjectIdentifier) {
objectLock := globalNSMutex.NewNSLock(bucket, obj.ObjectName)
objectLock.Lock()
defer objectLock.Unlock()
defer wg.Done()
dErr := objectAPI.DeleteObject(bucket, obj.ObjectName)
if dErr != nil {
dErrs[i] = dErr
@@ -318,6 +327,13 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
// Write success response.
writeSuccessResponseXML(w, encodedSuccessResponse)
// Get host and port from Request.RemoteAddr failing which
// fill them with empty strings.
host, port, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
host, port = "", ""
}
// Notify deleted event for objects.
for _, dobj := range deletedObjects {
eventNotify(eventData{
@@ -326,9 +342,10 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
ObjInfo: ObjectInfo{
Name: dobj.ObjectName,
},
ReqParams: map[string]string{
"sourceIPAddress": r.RemoteAddr,
},
ReqParams: extractReqParams(r),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
})
}
}
@@ -357,19 +374,26 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
vars := mux.Vars(r)
bucket := vars["bucket"]
// Validate if incoming location constraint is valid, reject
// requests which do not follow valid region requirements.
if s3Error := isValidLocationConstraint(r); s3Error != ErrNone {
// Parse incoming location constraint.
location, s3Error := parseLocationConstraint(r)
if s3Error != ErrNone {
writeErrorResponse(w, s3Error, r.URL)
return
}
// Validate if location sent by the client is valid, reject
// requests which do not follow valid region requirements.
if !isValidLocation(location) {
writeErrorResponse(w, ErrInvalidRegion, r.URL)
return
}
bucketLock := globalNSMutex.NewNSLock(bucket, "")
bucketLock.Lock()
defer bucketLock.Unlock()
// Proceed to creating a bucket.
err := objectAPI.MakeBucket(bucket)
err := objectAPI.MakeBucketWithLocation(bucket, "")
if err != nil {
errorIf(err, "Unable to create a bucket.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
@@ -438,14 +462,25 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
defer fileBody.Close()
bucket := mux.Vars(r)["bucket"]
formValues["Bucket"] = bucket
formValues.Set("Bucket", bucket)
if fileName != "" && strings.Contains(formValues["Key"], "${filename}") {
if fileName != "" && strings.Contains(formValues.Get("Key"), "${filename}") {
// S3 feature to replace ${filename} found in Key form field
// by the filename attribute passed in multipart
formValues["Key"] = strings.Replace(formValues["Key"], "${filename}", fileName, -1)
formValues.Set("Key", strings.Replace(formValues.Get("Key"), "${filename}", fileName, -1))
}
object := formValues.Get("Key")
successRedirect := formValues.Get("success_action_redirect")
successStatus := formValues.Get("success_action_status")
var redirectURL *url.URL
if successRedirect != "" {
redirectURL, err = url.Parse(successRedirect)
if err != nil {
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
}
}
object := formValues["Key"]
// Verify policy signature.
apiErr := doesPolicySignatureMatch(formValues)
@@ -454,7 +489,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
policyBytes, err := base64.StdEncoding.DecodeString(formValues["Policy"])
policyBytes, err := base64.StdEncoding.DecodeString(formValues.Get("Policy"))
if err != nil {
writeErrorResponse(w, ErrMalformedPOSTRequest, r.URL)
return
@@ -482,7 +517,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
if fileSize > lengthRange.Max || fileSize > maxObjectSize {
if fileSize > lengthRange.Max || isMaxObjectSize(fileSize) {
errorIf(err, "Unable to create object.")
writeErrorResponse(w, toAPIErrorCode(errDataTooLarge), r.URL)
return
@@ -490,8 +525,12 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
}
// Extract metadata to be saved from received Form.
metadata := extractMetadataFromForm(formValues)
metadata, err := extractMetadataFromHeader(formValues)
if err != nil {
errorIf(err, "found invalid http request header")
writeErrorResponse(w, ErrInternalError, r.URL)
return
}
sha256sum := ""
objectLock := globalNSMutex.NewNSLock(bucket, object)
@@ -504,50 +543,49 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
w.Header().Set("ETag", "\""+objInfo.MD5Sum+"\"")
w.Header().Set("ETag", `"`+objInfo.ETag+`"`)
w.Header().Set("Location", getObjectLocation(bucket, object))
successRedirect := formValues[http.CanonicalHeaderKey("success_action_redirect")]
successStatus := formValues[http.CanonicalHeaderKey("success_action_status")]
if successStatus == "" && successRedirect == "" {
writeSuccessNoContent(w)
} else {
if successRedirect != "" {
redirectURL := successRedirect + "?" + fmt.Sprintf("bucket=%s&key=%s&etag=%s",
bucket,
getURLEncodedName(object),
getURLEncodedName("\""+objInfo.MD5Sum+"\""))
writeRedirectSeeOther(w, redirectURL)
} else {
// Decide what http response to send depending on success_action_status parameter
switch successStatus {
case "201":
resp := encodeResponse(PostResponse{
Bucket: bucket,
Key: object,
ETag: "\"" + objInfo.MD5Sum + "\"",
Location: getObjectLocation(bucket, object),
})
writeResponse(w, http.StatusCreated, resp, "application/xml")
case "200":
writeSuccessResponseHeadersOnly(w)
default:
writeSuccessNoContent(w)
}
}
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
host, port = "", ""
}
// Notify object created event.
eventNotify(eventData{
Type: ObjectCreatedPost,
Bucket: bucket,
ObjInfo: objInfo,
ReqParams: map[string]string{
"sourceIPAddress": r.RemoteAddr,
},
defer eventNotify(eventData{
Type: ObjectCreatedPost,
Bucket: objInfo.Bucket,
ObjInfo: objInfo,
ReqParams: extractReqParams(r),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
})
if successRedirect != "" {
// Replace raw query params..
redirectURL.RawQuery = getRedirectPostRawQuery(objInfo)
writeRedirectSeeOther(w, redirectURL.String())
return
}
// Decide what http response to send depending on success_action_status parameter
switch successStatus {
case "201":
resp := encodeResponse(PostResponse{
Bucket: objInfo.Bucket,
Key: objInfo.Name,
ETag: `"` + objInfo.ETag + `"`,
Location: getObjectLocation(objInfo.Bucket, objInfo.Name),
})
writeResponse(w, http.StatusCreated, resp, "application/xml")
case "200":
writeSuccessResponseHeadersOnly(w)
default:
writeSuccessNoContent(w)
}
}
// HeadBucketHandler - HEAD Bucket
@@ -615,12 +653,21 @@ func (api objectAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http.
// Delete bucket access policy, if present - ignore any errors.
_ = removeBucketPolicy(bucket, objectAPI)
// Notify all peers (including self) to update in-memory state
S3PeersUpdateBucketPolicy(bucket, policyChange{true, nil})
// Delete notification config, if present - ignore any errors.
_ = removeNotificationConfig(bucket, objectAPI)
// Notify all peers (including self) to update in-memory state
S3PeersUpdateBucketNotification(bucket, nil)
// Delete listener config, if present - ignore any errors.
_ = removeListenerConfig(bucket, objectAPI)
// Notify all peers (including self) to update in-memory state
S3PeersUpdateBucketListener(bucket, []listenerConfig{})
// Write success response.
writeSuccessNoContent(w)
}
+34 -1
View File
@@ -68,7 +68,7 @@ func testGetBucketLocationHandler(obj ObjectLayer, instanceType, bucketName stri
locationResponse: []byte(""),
errorResponse: APIErrorResponse{
Resource: "/" + bucketName + "/",
Code: "InvalidAccessKeyID",
Code: "InvalidAccessKeyId",
Message: "The access key ID you provided does not exist in our records.",
},
shouldPass: false,
@@ -771,3 +771,36 @@ func testAPIDeleteMultipleObjectsHandler(obj ObjectLayer, instanceType, bucketNa
// `ExecObjectLayerAPINilTest` manages the operation.
ExecObjectLayerAPINilTest(t, nilBucket, nilObject, instanceType, apiRouter, nilReq)
}
func TestIsBucketActionAllowed(t *testing.T) {
ExecObjectLayerAPITest(t, testIsBucketActionAllowedHandler, []string{"BucketLocation"})
}
func testIsBucketActionAllowedHandler(obj ObjectLayer, instanceType, bucketName string, apiRouter http.Handler,
credentials credential, t *testing.T) {
testCases := []struct {
// input.
action string
bucket string
prefix string
isGlobalPoliciesNil bool
// flag indicating whether the test should pass.
shouldPass bool
}{
{"s3:GetBucketLocation", "mybucket", "abc", true, false},
{"s3:ListObject", "mybucket", "abc", false, false},
}
for i, testCase := range testCases {
if testCase.isGlobalPoliciesNil {
globalBucketPolicies = nil
} else {
initBucketPolicies(obj)
}
isAllowed := isBucketActionAllowed(testCase.action, testCase.bucket, testCase.prefix)
if isAllowed != testCase.shouldPass {
t.Errorf("Case %d: Expected the response status to be `%t`, but instead found `%t`", i+1, testCase.shouldPass, isAllowed)
}
}
}
+24 -5
View File
@@ -94,6 +94,10 @@ const (
ObjectCreatedCompleteMultipartUpload
// ObjectRemovedDelete is s3:ObjectRemoved:Delete
ObjectRemovedDelete
// ObjectAccessedGet is s3:ObjectAccessed:Get
ObjectAccessedGet
// ObjectAccessedHead is s3:ObjectAccessed:Head
ObjectAccessedHead
)
// Stringer interface for event name.
@@ -109,6 +113,10 @@ func (eventName EventName) String() string {
return "s3:ObjectCreated:CompleteMultipartUpload"
case ObjectRemovedDelete:
return "s3:ObjectRemoved:Delete"
case ObjectAccessedGet:
return "s3:ObjectAccessed:Get"
case ObjectAccessedHead:
return "s3:ObjectAccessed:Head"
default:
return "s3:Unknown"
}
@@ -128,11 +136,13 @@ type bucketMeta struct {
// Notification event object metadata.
type objectMeta struct {
Key string `json:"key"`
Size int64 `json:"size,omitempty"`
ETag string `json:"eTag,omitempty"`
VersionID string `json:"versionId,omitempty"`
Sequencer string `json:"sequencer"`
Key string `json:"key"`
Size int64 `json:"size,omitempty"`
ETag string `json:"eTag,omitempty"`
ContentType string `json:"contentType,omitempty"`
UserDefined map[string]string `json:"userDefined,omitempty"`
VersionID string `json:"versionId,omitempty"`
Sequencer string `json:"sequencer"`
}
const (
@@ -168,6 +178,14 @@ const (
eventVersion = "2.0"
)
// sourceInfo represents information on the client that triggered the
// event notification.
type sourceInfo struct {
Host string `json:"host"`
Port string `json:"port"`
UserAgent string `json:"userAgent"`
}
// NotificationEvent represents an Amazon an S3 bucket notification event.
type NotificationEvent struct {
EventVersion string `json:"eventVersion"`
@@ -179,6 +197,7 @@ type NotificationEvent struct {
RequestParameters map[string]string `json:"requestParameters"`
ResponseElements map[string]string `json:"responseElements"`
S3 eventMeta `json:"s3"`
Source sourceInfo `json:"source"`
}
// Represents the minio sqs type and account id's.
+12 -6
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -204,15 +204,21 @@ func writeNotification(w http.ResponseWriter, notification map[string][]Notifica
if err != nil {
return err
}
// https://github.com/containous/traefik/issues/560
// https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
//
// Proxies might buffer the connection to avoid this we
// need the proper MIME type before writing to client.
// This MIME header tells the proxies to avoid buffering
w.Header().Set("Content-Type", "text/event-stream")
// Add additional CRLF characters for client to
// differentiate the individual events properly.
_, err = w.Write(append(notificationBytes, crlf...))
// Make sure we have flushed, this would set Transfer-Encoding: chunked.
w.(http.Flusher).Flush()
if err != nil {
return err
}
return nil
return err
}
// CRLF character used for chunked transfer in accordance with HTTP standards.
@@ -288,7 +294,7 @@ func (api objectAPIHandlers) ListenBucketNotificationHandler(w http.ResponseWrit
return
}
accountID := fmt.Sprintf("%d", time.Now().UTC().UnixNano())
accountID := fmt.Sprintf("%d", UTCNow().UnixNano())
accountARN := fmt.Sprintf(
"%s:%s:%s:%s-%s",
minioTopic,
+2 -1
View File
@@ -185,7 +185,7 @@ func testGetBucketNotificationHandler(obj ObjectLayer, instanceType, bucketName
filterRules := []filterRule{
{
Name: "prefix",
Value: globalMinioDefaultOwnerID,
Value: "minio",
},
{
Name: "suffix",
@@ -268,6 +268,7 @@ func testListenBucketNotificationNilHandler(obj ObjectLayer, instanceType, bucke
[]string{"*.jpg"}, []string{
"s3:ObjectCreated:*",
"s3:ObjectRemoved:*",
"s3:ObjectAccessed:*",
}), 0, nil, credentials.AccessKey, credentials.SecretKey)
if tErr != nil {
t.Fatalf("%s: Failed to create HTTP testRequest for ListenBucketNotification: <ERROR> %v", instanceType, tErr)
+61 -31
View File
@@ -16,7 +16,12 @@
package cmd
import "strings"
import (
"errors"
"strings"
"github.com/minio/minio-go/pkg/set"
)
// List of valid event types.
var suppportedEventTypes = map[string]struct{}{
@@ -29,6 +34,9 @@ var suppportedEventTypes = map[string]struct{}{
// Object removed event types.
"s3:ObjectRemoved:*": {},
"s3:ObjectRemoved:Delete": {},
"s3:ObjectAccessed:Get": {},
"s3:ObjectAccessed:Head": {},
"s3:ObjectAccessed:*": {},
}
// checkEvent - checks if an event is supported.
@@ -104,20 +112,21 @@ func checkARN(arn, arnType string) APIErrorCode {
if !strings.HasPrefix(arn, arnType) {
return ErrARNNotification
}
if !strings.HasPrefix(arn, arnType+serverConfig.GetRegion()+":") {
return ErrRegionNotification
}
account := strings.SplitN(strings.TrimPrefix(arn, arnType+serverConfig.GetRegion()+":"), ":", 2)
switch len(account) {
case 1:
// This means ARN is malformed, account should have min of 2elements.
strs := strings.SplitN(arn, ":", -1)
if len(strs) != 6 {
return ErrARNNotification
case 2:
// Account topic id or topic name cannot be empty.
if account[0] == "" || account[1] == "" {
return ErrARNNotification
}
if serverConfig.GetRegion() != "" {
region := strs[3]
if region != serverConfig.GetRegion() {
return ErrRegionNotification
}
}
accountID := strs[4]
resource := strs[5]
if accountID == "" || resource == "" {
return ErrARNNotification
}
return ErrNone
}
@@ -135,6 +144,9 @@ func isValidQueueID(queueARN string) bool {
if isAMQPQueue(sqsARN) { // AMQP eueue.
amqpN := serverConfig.Notify.GetAMQPByID(sqsARN.AccountID)
return amqpN.Enable && amqpN.URL != ""
} else if isMQTTQueue(sqsARN) {
mqttN := serverConfig.Notify.GetMQTTByID(sqsARN.AccountID)
return mqttN.Enable && mqttN.Broker != ""
} else if isNATSQueue(sqsARN) {
natsN := serverConfig.Notify.GetNATSByID(sqsARN.AccountID)
return natsN.Enable && natsN.Address != ""
@@ -148,6 +160,10 @@ func isValidQueueID(queueARN string) bool {
pgN := serverConfig.Notify.GetPostgreSQLByID(sqsARN.AccountID)
// Postgres can work with only default conn. info.
return pgN.Enable
} else if isMySQLQueue(sqsARN) {
msqlN := serverConfig.Notify.GetMySQLByID(sqsARN.AccountID)
// Mysql can work with only default conn. info.
return msqlN.Enable
} else if isKafkaQueue(sqsARN) {
kafkaN := serverConfig.Notify.GetKafkaByID(sqsARN.AccountID)
return (kafkaN.Enable && len(kafkaN.Brokers) > 0 &&
@@ -200,16 +216,14 @@ func validateQueueConfigs(queueConfigs []queueConfig) APIErrorCode {
// Check all the queue configs for any duplicates.
func checkDuplicateQueueConfigs(configs []queueConfig) APIErrorCode {
var queueConfigARNS []string
queueConfigARNS := set.NewStringSet()
// Navigate through each configs and count the entries.
for _, config := range configs {
queueConfigARNS = append(queueConfigARNS, config.QueueARN)
queueConfigARNS.Add(config.QueueARN)
}
// Check if there are any duplicate counts.
if err := checkDuplicateStrings(queueConfigARNS); err != nil {
errorIf(err, "Invalid queue configs found.")
if len(queueConfigARNS) != len(configs) {
return ErrOverlappingConfigs
}
@@ -240,34 +254,50 @@ func validateNotificationConfig(nConfig notificationConfig) APIErrorCode {
// Unmarshals input value of AWS ARN format into minioSqs object.
// Returned value represents minio sqs types, currently supported are
// - amqp
// - mqtt
// - nats
// - elasticsearch
// - redis
// - postgresql
// - mysql
// - kafka
// - webhook
func unmarshalSqsARN(queueARN string) (mSqs arnSQS) {
mSqs = arnSQS{}
if !strings.HasPrefix(queueARN, minioSqs+serverConfig.GetRegion()+":") {
return mSqs
strs := strings.SplitN(queueARN, ":", -1)
if len(strs) != 6 {
return
}
sqsType := strings.TrimPrefix(queueARN, minioSqs+serverConfig.GetRegion()+":")
switch {
case strings.HasSuffix(sqsType, queueTypeAMQP):
if serverConfig.GetRegion() != "" {
region := strs[3]
if region != serverConfig.GetRegion() {
return
}
}
sqsType := strs[5]
switch sqsType {
case queueTypeAMQP:
mSqs.Type = queueTypeAMQP
case strings.HasSuffix(sqsType, queueTypeNATS):
case queueTypeMQTT:
mSqs.Type = queueTypeMQTT
case queueTypeNATS:
mSqs.Type = queueTypeNATS
case strings.HasSuffix(sqsType, queueTypeElastic):
case queueTypeElastic:
mSqs.Type = queueTypeElastic
case strings.HasSuffix(sqsType, queueTypeRedis):
case queueTypeRedis:
mSqs.Type = queueTypeRedis
case strings.HasSuffix(sqsType, queueTypePostgreSQL):
case queueTypePostgreSQL:
mSqs.Type = queueTypePostgreSQL
case strings.HasSuffix(sqsType, queueTypeKafka):
case queueTypeMySQL:
mSqs.Type = queueTypeMySQL
case queueTypeKafka:
mSqs.Type = queueTypeKafka
case strings.HasSuffix(sqsType, queueTypeWebhook):
case queueTypeWebhook:
mSqs.Type = queueTypeWebhook
default:
errorIf(errors.New("invalid SQS type"), "SQS type: %s", sqsType)
} // Add more queues here.
mSqs.AccountID = strings.TrimSuffix(sqsType, ":"+mSqs.Type)
return mSqs
mSqs.AccountID = strs[4]
return
}
+65 -10
View File
@@ -259,11 +259,6 @@ func TestQueueARN(t *testing.T) {
queueARN: "arn:minio:sns:us-east-1:1:listen",
errCode: ErrARNNotification,
},
// Invalid region 'us-west-1' in queue arn.
{
queueARN: "arn:minio:sqs:us-west-1:1:redis",
errCode: ErrRegionNotification,
},
// Invalid queue name empty in queue arn.
{
queueARN: "arn:minio:sqs:us-east-1:1:",
@@ -298,6 +293,37 @@ func TestQueueARN(t *testing.T) {
t.Errorf("Test %d: Expected \"%d\", got \"%d\"", i+1, testCase.errCode, errCode)
}
}
// Test when server region is set.
rootPath, err = newTestConfig("us-east-1")
if err != nil {
t.Fatalf("unable initialize config file, %s", err)
}
defer removeAll(rootPath)
testCases = []struct {
queueARN string
errCode APIErrorCode
}{
// Incorrect region should produce error.
{
queueARN: "arn:minio:sqs:us-west-1:1:webhook",
errCode: ErrRegionNotification,
},
// Correct region should not produce error.
{
queueARN: "arn:minio:sqs:us-east-1:1:webhook",
errCode: ErrNone,
},
}
// Validate all tests for queue arn.
for i, testCase := range testCases {
errCode := checkQueueARN(testCase.queueARN)
if testCase.errCode != errCode {
t.Errorf("Test %d: Expected \"%d\", got \"%d\"", i+1, testCase.errCode, errCode)
}
}
}
// Test unmarshal queue arn.
@@ -332,16 +358,16 @@ func TestUnmarshalSQSARN(t *testing.T) {
queueARN: "arn:minio:sqs:us-east-1:1:amqp",
Type: "amqp",
},
// Valid mqtt queue arn.
{
queueARN: "arn:minio:sqs:us-east-1:1:mqtt",
Type: "mqtt",
},
// Invalid empty queue arn.
{
queueARN: "",
Type: "",
},
// Invalid region 'us-west-1' in queue arn.
{
queueARN: "arn:minio:sqs:us-west-1:1:redis",
Type: "",
},
// Partial queue arn.
{
queueARN: "arn:minio:sqs:",
@@ -361,4 +387,33 @@ func TestUnmarshalSQSARN(t *testing.T) {
}
}
// Test when the server region is set.
rootPath, err = newTestConfig("us-east-1")
if err != nil {
t.Fatalf("unable initialize config file, %s", err)
}
defer removeAll(rootPath)
testCases = []struct {
queueARN string
Type string
}{
// Incorrect region in ARN returns empty mSqs.Type
{
queueARN: "arn:minio:sqs:us-west-1:1:webhook",
Type: "",
},
// Correct regionin ARN returns valid mSqs.Type
{
queueARN: "arn:minio:sqs:us-east-1:1:webhook",
Type: "webhook",
},
}
for i, testCase := range testCases {
mSqs := unmarshalSqsARN(testCase.queueARN)
if testCase.Type != mSqs.Type {
t.Errorf("Test %d: Expected \"%s\", got \"%s\"", i+1, testCase.Type, mSqs.Type)
}
}
}
+1 -1
View File
@@ -251,7 +251,7 @@ func testPutBucketPolicyHandler(obj ObjectLayer, instanceType, bucketName string
initBucketPolicies(obj)
bucketName1 := fmt.Sprintf("%s-1", bucketName)
if err := obj.MakeBucket(bucketName1); err != nil {
if err := obj.MakeBucketWithLocation(bucketName1, ""); err != nil {
t.Fatal(err)
}
+71 -122
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,150 +17,99 @@
package cmd
import (
"crypto/tls"
"crypto/x509"
"encoding/pem"
"errors"
"fmt"
"io/ioutil"
"os"
"path/filepath"
)
// createCertsPath create certs path.
func createCertsPath() error {
certsPath, err := getCertsPath()
if err != nil {
return err
}
if err := os.MkdirAll(certsPath, 0700); err != nil {
return err
}
rootCAsPath := filepath.Join(certsPath, globalMinioCertsCADir)
return os.MkdirAll(rootCAsPath, 0700)
}
// getCertsPath get certs path.
func getCertsPath() (string, error) {
var certsPath string
configDir, err := getConfigPath()
if err != nil {
return "", err
}
certsPath = filepath.Join(configDir, globalMinioCertsDir)
return certsPath, nil
}
// mustGetCertsPath must get certs path.
func mustGetCertsPath() string {
certsPath, err := getCertsPath()
fatalIf(err, "Failed to get certificate path.")
return certsPath
}
// mustGetCertFile must get cert file.
func mustGetCertFile() string {
return filepath.Join(mustGetCertsPath(), globalMinioCertFile)
}
// mustGetKeyFile must get key file.
func mustGetKeyFile() string {
return filepath.Join(mustGetCertsPath(), globalMinioKeyFile)
}
// mustGetCAFiles must get the list of the CA certificates stored in minio config dir
func mustGetCAFiles() (caCerts []string) {
CAsDir := filepath.Join(mustGetCertsPath(), globalMinioCertsCADir)
caFiles, _ := ioutil.ReadDir(CAsDir)
for _, cert := range caFiles {
caCerts = append(caCerts, filepath.Join(CAsDir, cert.Name()))
}
return
}
// mustGetSystemCertPool returns empty cert pool in case of error (windows)
func mustGetSystemCertPool() *x509.CertPool {
pool, err := x509.SystemCertPool()
if err != nil {
return x509.NewCertPool()
}
return pool
}
// isCertFileExists verifies if cert file exists, returns true if
// found, false otherwise.
func isCertFileExists() bool {
st, e := os.Stat(filepath.Join(mustGetCertsPath(), globalMinioCertFile))
// If file exists and is regular return true.
if e == nil && st.Mode().IsRegular() {
return true
}
return false
}
// isKeyFileExists verifies if key file exists, returns true if found,
// false otherwise.
func isKeyFileExists() bool {
st, e := os.Stat(filepath.Join(mustGetCertsPath(), globalMinioKeyFile))
// If file exists and is regular return true.
if e == nil && st.Mode().IsRegular() {
return true
}
return false
}
// isSSL - returns true with both cert and key exists.
func isSSL() bool {
return isCertFileExists() && isKeyFileExists()
}
// Reads certificated file and returns a list of parsed certificates.
func readCertificateChain() ([]*x509.Certificate, error) {
bytes, err := ioutil.ReadFile(mustGetCertFile())
if err != nil {
func parsePublicCertFile(certFile string) (x509Certs []*x509.Certificate, err error) {
// Read certificate file.
var data []byte
if data, err = ioutil.ReadFile(certFile); err != nil {
return nil, err
}
// Proceed to parse the certificates.
return parseCertificateChain(bytes)
}
// Parses certificate chain, returns a list of parsed certificates.
func parseCertificateChain(bytes []byte) ([]*x509.Certificate, error) {
var certs []*x509.Certificate
var block *pem.Block
current := bytes
// Parse all certs in the chain.
current := data
for len(current) > 0 {
block, current = pem.Decode(current)
if block == nil {
return nil, errors.New("Could not PEM block")
var pemBlock *pem.Block
if pemBlock, current = pem.Decode(current); pemBlock == nil {
return nil, fmt.Errorf("Could not read PEM block from file %s", certFile)
}
// Parse the decoded certificate.
cert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
var x509Cert *x509.Certificate
if x509Cert, err = x509.ParseCertificate(pemBlock.Bytes); err != nil {
return nil, err
}
certs = append(certs, cert)
x509Certs = append(x509Certs, x509Cert)
}
return certs, nil
if len(x509Certs) == 0 {
return nil, fmt.Errorf("Empty public certificate file %s", certFile)
}
return x509Certs, nil
}
// loadRootCAs fetches CA files provided in minio config and adds them to globalRootCAs
// Currently under Windows, there is no way to load system + user CAs at the same time
func loadRootCAs() {
caFiles := mustGetCAFiles()
if len(caFiles) == 0 {
return
func getRootCAs(certsCAsDir string) (*x509.CertPool, error) {
// Get all CA file names.
var caFiles []string
fis, err := ioutil.ReadDir(certsCAsDir)
if err != nil {
return nil, err
}
// Get system cert pool, and empty cert pool under Windows because it is not supported
globalRootCAs = mustGetSystemCertPool()
for _, fi := range fis {
caFiles = append(caFiles, filepath.Join(certsCAsDir, fi.Name()))
}
if len(caFiles) == 0 {
return nil, nil
}
rootCAs, err := x509.SystemCertPool()
if err != nil {
// In some systems like Windows, system cert pool is not supported.
// Hence we create a new cert pool.
rootCAs = x509.NewCertPool()
}
// Load custom root CAs for client requests
for _, caFile := range caFiles {
caCert, err := ioutil.ReadFile(caFile)
if err != nil {
fatalIf(err, "Unable to load a CA file")
return nil, err
}
globalRootCAs.AppendCertsFromPEM(caCert)
rootCAs.AppendCertsFromPEM(caCert)
}
return rootCAs, nil
}
func getSSLConfig() (x509Certs []*x509.Certificate, rootCAs *x509.CertPool, tlsCert *tls.Certificate, secureConn bool, err error) {
if !(isFile(getPublicCertFile()) && isFile(getPrivateKeyFile())) {
return nil, nil, nil, false, nil
}
if x509Certs, err = parsePublicCertFile(getPublicCertFile()); err != nil {
return nil, nil, nil, false, err
}
var cert tls.Certificate
if cert, err = tls.LoadX509KeyPair(getPublicCertFile(), getPrivateKeyFile()); err != nil {
return nil, nil, nil, false, err
}
tlsCert = &cert
if rootCAs, err = getRootCAs(getCADir()); err != nil {
return nil, nil, nil, false, err
}
secureConn = true
return x509Certs, rootCAs, tlsCert, secureConn, nil
}
+203 -55
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,52 +17,86 @@
package cmd
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strings"
"runtime"
"testing"
)
// Make sure we have a valid certs path.
func TestGetCertsPath(t *testing.T) {
path, err := getCertsPath()
func createTempFile(prefix, content string) (tempFile string, err error) {
var tmpfile *os.File
if tmpfile, err = ioutil.TempFile("", prefix); err != nil {
return tempFile, err
}
if _, err = tmpfile.Write([]byte(content)); err != nil {
return tempFile, err
}
if err = tmpfile.Close(); err != nil {
return tempFile, err
}
tempFile = tmpfile.Name()
return tempFile, err
}
func TestParsePublicCertFile(t *testing.T) {
tempFile1, err := createTempFile("public-cert-file", "")
if err != nil {
t.Error(err)
}
if path == "" {
t.Errorf("expected path to not be an empty string, got: '%s'", path)
}
// Ensure it contains some sort of path separator.
if !strings.ContainsRune(path, os.PathSeparator) {
t.Errorf("expected path to contain file separator")
}
// It should also be an absolute path.
if !filepath.IsAbs(path) {
t.Errorf("expected path to be an absolute path")
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile1)
// This will error if something goes wrong, so just call it.
mustGetCertsPath()
}
// Ensure that the certificate and key file getters contain their respective
// file name and endings.
func TestGetFiles(t *testing.T) {
file := mustGetCertFile()
if !strings.Contains(file, globalMinioCertFile) {
t.Errorf("CertFile does not contain %s", globalMinioCertFile)
tempFile2, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBA-some-junk-Q4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile2)
file = mustGetKeyFile()
if !strings.Contains(file, globalMinioKeyFile) {
t.Errorf("KeyFile does not contain %s", globalMinioKeyFile)
tempFile3, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBAabababababaQ4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Unable to create temporary file. %v", err)
}
}
defer os.Remove(tempFile3)
// Parses .crt file contents
func TestParseCertificateChain(t *testing.T) {
// given
cert := `-----BEGIN CERTIFICATE-----
tempFile4, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
@@ -77,35 +111,149 @@ JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`
// when
certs, err := parseCertificateChain([]byte(cert))
// then
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Could not parse certificate: %s", err)
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile4)
tempFile5, err := createTempFile("public-cert-file",
`-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBAoTBU1pbmlvMQ4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICdTCCAd4CCQCO5G/W1xcE9TANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJa
WTEOMAwGA1UECBMFTWluaW8xETAPBgNVBAcTCEludGVybmV0MQ4wDAYDVQQKEwVN
aW5pbzEOMAwGA1UECxMFTWluaW8xDjAMBgNVBAMTBU1pbmlvMR0wGwYJKoZIhvcN
AQkBFg50ZXN0c0BtaW5pby5pbzAeFw0xNjEwMTQxMTM0MjJaFw0xNzEwMTQxMTM0
MjJaMH8xCzAJBgNVBAYTAlpZMQ4wDAYDVQQIEwVNaW5pbzERMA8GA1UEBxMISW50
ZXJuZXQxDjAMBgNVBAoTBU1pbmlvMQ4wDAYDVQQLEwVNaW5pbzEOMAwGA1UEAxMF
TWluaW8xHTAbBgkqhkiG9w0BCQEWDnRlc3RzQG1pbmlvLmlvMIGfMA0GCSqGSIb3
DQEBAQUAA4GNADCBiQKBgQDwNUYB/Sj79WsUE8qnXzzh2glSzWxUE79sCOpQYK83
HWkrl5WxlG8ZxDR1IQV9Ex/lzigJu8G+KXahon6a+3n5GhNrYRe5kIXHQHz0qvv4
aMulqlnYpvSfC83aaO9GVBtwXS/O4Nykd7QBg4nZlazVmsGk7POOjhpjGShRsqpU
JwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBALqjOA6bD8BEl7hkQ8XwX/owSAL0URDe
nUfCOsXgIIAqgw4uTCLOfCJVZNKmRT+KguvPAQ6Z80vau2UxPX5Q2Q+OHXDRrEnK
FjqSBgLP06Qw7a++bshlWGTt5bHWOneW3EQikedckVuIKPkOCib9yGi4VmBBjdFE
M9ofSEt/bdRD
-----END CERTIFICATE-----`)
if err != nil {
t.Fatalf("Unable to create temporary file. %v", err)
}
defer os.Remove(tempFile5)
nonexistentErr := fmt.Errorf("open nonexistent-file: no such file or directory")
if runtime.GOOS == "windows" {
// Below concatenation is done to get rid of goline error
// "error strings should not be capitalized or end with punctuation or a newline"
nonexistentErr = fmt.Errorf("open nonexistent-file:" + " The system cannot find the file specified.")
}
if len(certs) != 1 {
t.Fatalf("Expected number of certificates in chain was 1, actual: %d", len(certs))
testCases := []struct {
certFile string
expectedResultLen int
expectedErr error
}{
{"nonexistent-file", 0, nonexistentErr},
{tempFile1, 0, fmt.Errorf("Empty public certificate file %s", tempFile1)},
{tempFile2, 0, fmt.Errorf("Could not read PEM block from file %s", tempFile2)},
{tempFile3, 0, fmt.Errorf("asn1: structure error: sequence tag mismatch")},
{tempFile4, 1, nil},
{tempFile5, 2, nil},
}
if certs[0].Subject.CommonName != "Minio" {
t.Fatalf("Expected Subject.CommonName was Minio, actual: %s", certs[0].Subject.CommonName)
for _, testCase := range testCases {
certs, err := parsePublicCertFile(testCase.certFile)
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
} else if err == nil {
t.Fatalf("error: expected = %v, got = <nil>", testCase.expectedErr)
} else if testCase.expectedErr.Error() != err.Error() {
t.Fatalf("error: expected = %v, got = %v", testCase.expectedErr, err)
}
if len(certs) != testCase.expectedResultLen {
t.Fatalf("certs: expected = %v, got = %v", testCase.expectedResultLen, len(certs))
}
}
}
// Parses invalid .crt file contents and returns error
func TestParseInvalidCertificateChain(t *testing.T) {
// given
cert := `This is now valid certificate`
func TestGetRootCAs(t *testing.T) {
emptydir, err := ioutil.TempDir("", "test-get-root-cas")
if err != nil {
t.Fatalf("Unable create temp directory. %v", emptydir)
}
defer os.RemoveAll(emptydir)
// when
_, err := parseCertificateChain([]byte(cert))
dir1, err := ioutil.TempDir("", "test-get-root-cas")
if err != nil {
t.Fatalf("Unable create temp directory. %v", dir1)
}
defer os.RemoveAll(dir1)
if err = os.Mkdir(filepath.Join(dir1, "empty-dir"), 0755); err != nil {
t.Fatalf("Unable create empty dir. %v", err)
}
// then
if err == nil {
t.Fatalf("Expected error but none occurred")
dir2, err := ioutil.TempDir("", "test-get-root-cas")
if err != nil {
t.Fatalf("Unable create temp directory. %v", dir2)
}
defer os.RemoveAll(dir2)
if err = ioutil.WriteFile(filepath.Join(dir2, "empty-file"), []byte{}, 0644); err != nil {
t.Fatalf("Unable create test file. %v", err)
}
nonexistentErr := fmt.Errorf("open nonexistent-dir: no such file or directory")
if runtime.GOOS == "windows" {
// Below concatenation is done to get rid of goline error
// "error strings should not be capitalized or end with punctuation or a newline"
nonexistentErr = fmt.Errorf("open nonexistent-dir:" + " The system cannot find the file specified.")
}
err1 := fmt.Errorf("read %s: is a directory", filepath.Join(dir1, "empty-dir"))
if runtime.GOOS == "windows" {
// Below concatenation is done to get rid of goline error
// "error strings should not be capitalized or end with punctuation or a newline"
err1 = fmt.Errorf("read %s:"+" The handle is invalid.", filepath.Join(dir1, "empty-dir"))
}
testCases := []struct {
certCAsDir string
expectedErr error
}{
{"nonexistent-dir", nonexistentErr},
{dir1, err1},
{emptydir, nil},
{dir2, nil},
}
for _, testCase := range testCases {
_, err := getRootCAs(testCase.certCAsDir)
if testCase.expectedErr == nil {
if err != nil {
t.Fatalf("error: expected = <nil>, got = %v", err)
}
} else if err == nil {
t.Fatalf("error: expected = %v, got = <nil>", testCase.expectedErr)
} else if testCase.expectedErr.Error() != err.Error() {
t.Fatalf("error: expected = %v, got = %v", testCase.expectedErr, err)
}
}
}
-64
View File
@@ -1,64 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"net"
"os"
"syscall"
)
// checkPortAvailability - check if given port is already in use.
// Note: The check method tries to listen on given port and closes it.
// It is possible to have a disconnected client in this tiny window of time.
func checkPortAvailability(port string) error {
network := [3]string{"tcp", "tcp4", "tcp6"}
for _, n := range network {
l, err := net.Listen(n, net.JoinHostPort("", port))
if err != nil {
if isAddrInUse(err) {
// Return error if another process is listening on the
// same port.
return err
}
// Ignore any other error (ex. EAFNOSUPPORT)
continue
}
// look for error so we don't have dangling connection
if err = l.Close(); err != nil {
return err
}
}
return nil
}
// Return true if err is "address already in use" error.
// syscall.EADDRINUSE is available on all OSes.
func isAddrInUse(err error) bool {
if opErr, ok := err.(*net.OpError); ok {
if sysErr, ok := opErr.Err.(*os.SyscallError); ok {
if errno, ok := sysErr.Err.(syscall.Errno); ok {
if errno == syscall.EADDRINUSE {
return true
}
}
}
}
return false
}
-54
View File
@@ -1,54 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"net"
"runtime"
"testing"
)
// Tests for port availability logic written for server startup sequence.
func TestCheckPortAvailability(t *testing.T) {
tests := []struct {
port string
}{
{getFreePort()},
{getFreePort()},
}
for _, test := range tests {
// This test should pass if the ports are available
err := checkPortAvailability(test.port)
if err != nil {
t.Fatalf("checkPortAvailability test failed for port: %s. Error: %v", test.port, err)
}
// Now use the ports and check again
ln, err := net.Listen("tcp", net.JoinHostPort("", test.port))
if err != nil {
t.Fail()
}
defer ln.Close()
err = checkPortAvailability(test.port)
// Skip if the os is windows due to https://github.com/golang/go/issues/7598
if err == nil && runtime.GOOS != globalWindowsOSName {
t.Fatalf("checkPortAvailability should fail for port: %s. Error: %v", test.port, err)
}
}
}
+117
View File
@@ -0,0 +1,117 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"errors"
"os"
"path/filepath"
"strings"
"time"
"github.com/minio/cli"
)
// Check for updates and print a notification message
func checkUpdate(mode string) {
// Its OK to ignore any errors during getUpdateInfo() here.
if older, downloadURL, err := getUpdateInfo(1*time.Second, mode); err == nil {
if updateMsg := computeUpdateMessage(downloadURL, older); updateMsg != "" {
log.Println(updateMsg)
}
}
}
func enableLoggers() {
fileLogTarget := serverConfig.Logger.GetFile()
if fileLogTarget.Enable {
err := InitFileLogger(&fileLogTarget)
fatalIf(err, "Unable to initialize file logger")
log.AddTarget(fileLogTarget)
}
consoleLogTarget := serverConfig.Logger.GetConsole()
if consoleLogTarget.Enable {
InitConsoleLogger(&consoleLogTarget)
}
log.SetConsoleTarget(consoleLogTarget)
}
func initConfig() {
// Config file does not exist, we create it fresh and return upon success.
if isFile(getConfigFile()) {
fatalIf(migrateConfig(), "Config migration failed.")
fatalIf(loadConfig(), "Unable to load config version: '%s'.", v19)
} else {
fatalIf(newConfig(), "Unable to initialize minio config for the first time.")
log.Println("Created minio configuration file successfully at " + getConfigDir())
}
}
func handleCommonCmdArgs(ctx *cli.Context) {
// Set configuration directory.
{
// Get configuration directory from command line argument.
configDir := ctx.String("config-dir")
if !ctx.IsSet("config-dir") && ctx.GlobalIsSet("config-dir") {
configDir = ctx.GlobalString("config-dir")
}
if configDir == "" {
fatalIf(errors.New("empty directory"), "Configuration directory cannot be empty.")
}
// Disallow relative paths, figure out absolute paths.
configDirAbs, err := filepath.Abs(configDir)
fatalIf(err, "Unable to fetch absolute path for config directory %s", configDir)
setConfigDir(configDirAbs)
}
}
func handleCommonEnvVars() {
// Start profiler if env is set.
if profiler := os.Getenv("_MINIO_PROFILER"); profiler != "" {
globalProfiler = startProfiler(profiler)
}
// Check if object cache is disabled.
globalXLObjCacheDisabled = strings.EqualFold(os.Getenv("_MINIO_CACHE"), "off")
accessKey := os.Getenv("MINIO_ACCESS_KEY")
secretKey := os.Getenv("MINIO_SECRET_KEY")
if accessKey != "" && secretKey != "" {
cred, err := createCredential(accessKey, secretKey)
fatalIf(err, "Invalid access/secret Key set in environment.")
// credential Envs are set globally.
globalIsEnvCreds = true
globalActiveCred = cred
}
if browser := os.Getenv("MINIO_BROWSER"); browser != "" {
browserFlag, err := ParseBrowserFlag(browser)
if err != nil {
fatalIf(errors.New("invalid value"), "Unknown value %s in MINIO_BROWSER environment variable.", browser)
}
// browser Envs are set globally, this does not represent
// if browser is turned off or on.
globalIsEnvBrowser = true
globalIsBrowserEnabled = bool(browserFlag)
}
}
+132
View File
@@ -0,0 +1,132 @@
/*
* Minio Cloud Storage, (C) 2015, 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"path/filepath"
"sync"
homedir "github.com/minio/go-homedir"
)
const (
// Default minio configuration directory where below configuration files/directories are stored.
defaultMinioConfigDir = ".minio"
// Minio configuration file.
minioConfigFile = "config.json"
// Directory contains below files/directories for HTTPS configuration.
certsDir = "certs"
// Directory contains all CA certificates other than system defaults for HTTPS.
certsCADir = "CAs"
// Public certificate file for HTTPS.
publicCertFile = "public.crt"
// Private key file for HTTPS.
privateKeyFile = "private.key"
)
// ConfigDir - configuration directory with locking.
type ConfigDir struct {
sync.Mutex
dir string
}
// Set - saves given directory as configuration directory.
func (config *ConfigDir) Set(dir string) {
config.Lock()
defer config.Unlock()
config.dir = dir
}
// Get - returns current configuration directory.
func (config *ConfigDir) Get() string {
config.Lock()
defer config.Unlock()
return config.dir
}
func (config *ConfigDir) getCertsDir() string {
return filepath.Join(config.Get(), certsDir)
}
// GetCADir - returns certificate CA directory.
func (config *ConfigDir) GetCADir() string {
return filepath.Join(config.getCertsDir(), certsCADir)
}
// Create - creates configuration directory tree.
func (config *ConfigDir) Create() error {
return mkdirAll(config.GetCADir(), 0700)
}
// GetMinioConfigFile - returns absolute path of config.json file.
func (config *ConfigDir) GetMinioConfigFile() string {
return filepath.Join(config.Get(), minioConfigFile)
}
// GetPublicCertFile - returns absolute path of public.crt file.
func (config *ConfigDir) GetPublicCertFile() string {
return filepath.Join(config.getCertsDir(), publicCertFile)
}
// GetPrivateKeyFile - returns absolute path of private.key file.
func (config *ConfigDir) GetPrivateKeyFile() string {
return filepath.Join(config.getCertsDir(), privateKeyFile)
}
func mustGetDefaultConfigDir() string {
homeDir, err := homedir.Dir()
fatalIf(err, "Unable to get home directory.")
return filepath.Join(homeDir, defaultMinioConfigDir)
}
var configDir = &ConfigDir{dir: mustGetDefaultConfigDir()}
func setConfigDir(dir string) {
configDir.Set(dir)
}
func getConfigDir() string {
return configDir.Get()
}
func getCADir() string {
return configDir.GetCADir()
}
func createConfigDir() error {
return configDir.Create()
}
func getConfigFile() string {
return configDir.GetMinioConfigFile()
}
func getPublicCertFile() string {
return configDir.GetPublicCertFile()
}
func getPrivateKeyFile() string {
return configDir.GetPrivateKeyFile()
}
+857 -316
View File
File diff suppressed because it is too large Load Diff
+89 -14
View File
@@ -17,6 +17,7 @@
package cmd
import (
"fmt"
"io/ioutil"
"os"
"testing"
@@ -31,7 +32,7 @@ func TestServerConfigMigrateV1(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
setConfigDir(rootPath)
// Create a V1 config json file and store it
configJSON := "{ \"version\":\"1\", \"accessKeyId\":\"abcde\", \"secretAccessKey\":\"abcdefgh\"}"
@@ -45,12 +46,12 @@ func TestServerConfigMigrateV1(t *testing.T) {
t.Fatal("Unexpected error: ", err)
}
// Check if config v1 is removed from filesystem
if _, err := os.Stat(configPath); err == nil || !os.IsNotExist(err) {
if _, err := osStat(configPath); err == nil || !os.IsNotExist(err) {
t.Fatal("Config V1 file is not purged")
}
// Initialize server config and check again if everything is fine
if err := loadConfig(credential{}); err != nil {
if err := loadConfig(); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
}
@@ -65,8 +66,8 @@ func TestServerConfigMigrateInexistentConfig(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
configPath := rootPath + "/" + globalMinioConfigFile
setConfigDir(rootPath)
configPath := rootPath + "/" + minioConfigFile
// Remove config file
if err := os.Remove(configPath); err != nil {
@@ -106,10 +107,29 @@ func TestServerConfigMigrateInexistentConfig(t *testing.T) {
if err := migrateV12ToV13(); err != nil {
t.Fatal("migrate v12 to v13 should succeed when no config file is found")
}
if err := migrateV13ToV14(); err != nil {
t.Fatal("migrate v13 to v14 should succeed when no config file is found")
}
if err := migrateV14ToV15(); err != nil {
t.Fatal("migrate v14 to v15 should succeed when no config file is found")
}
if err := migrateV15ToV16(); err != nil {
t.Fatal("migrate v15 to v16 should succeed when no config file is found")
}
if err := migrateV16ToV17(); err != nil {
t.Fatal("migrate v16 to v17 should succeed when no config file is found")
}
if err := migrateV17ToV18(); err != nil {
t.Fatal("migrate v17 to v18 should succeed when no config file is found")
}
if err := migrateV18ToV19(); err != nil {
t.Fatal("migrate v18 to v19 should succeed when no config file is found")
}
}
// Test if a config migration from v2 to v12 is successfully done
func TestServerConfigMigrateV2toV12(t *testing.T) {
// Test if a config migration from v2 to v19 is successfully done
func TestServerConfigMigrateV2toV19(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("Init Test config failed")
@@ -117,8 +137,8 @@ func TestServerConfigMigrateV2toV12(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
configPath := rootPath + "/" + globalMinioConfigFile
setConfigDir(rootPath)
configPath := rootPath + "/" + minioConfigFile
// Create a corrupted config file
if err := ioutil.WriteFile(configPath, []byte("{ \"version\":\"2\","), 0644); err != nil {
@@ -137,18 +157,19 @@ func TestServerConfigMigrateV2toV12(t *testing.T) {
if err := ioutil.WriteFile(configPath, []byte(configJSON), 0644); err != nil {
t.Fatal("Unexpected error: ", err)
}
// Fire a migrateConfig()
if err := migrateConfig(); err != nil {
t.Fatal("Unexpected error: ", err)
}
// Initialize server config and check again if everything is fine
if err := loadConfig(credential{}); err != nil {
if err := loadConfig(); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
// Check the version number in the upgraded config file
expectedVersion := globalMinioConfigVersion
expectedVersion := v19
if serverConfig.Version != expectedVersion {
t.Fatalf("Expect version "+expectedVersion+", found: %v", serverConfig.Version)
}
@@ -171,11 +192,11 @@ func TestServerConfigMigrateFaultyConfig(t *testing.T) {
// remove the root directory after the test ends.
defer removeAll(rootPath)
setGlobalConfigPath(rootPath)
configPath := rootPath + "/" + globalMinioConfigFile
setConfigDir(rootPath)
configPath := rootPath + "/" + minioConfigFile
// Create a corrupted config file
if err := ioutil.WriteFile(configPath, []byte("{ \"version\":\""), 0644); err != nil {
if err := ioutil.WriteFile(configPath, []byte("{ \"version\":\"2\", \"test\":"), 0644); err != nil {
t.Fatal("Unexpected error: ", err)
}
@@ -213,4 +234,58 @@ func TestServerConfigMigrateFaultyConfig(t *testing.T) {
if err := migrateV12ToV13(); err == nil {
t.Fatal("migrateConfigV12ToV13() should fail with a corrupted json")
}
if err := migrateV13ToV14(); err == nil {
t.Fatal("migrateConfigV13ToV14() should fail with a corrupted json")
}
if err := migrateV14ToV15(); err == nil {
t.Fatal("migrateConfigV14ToV15() should fail with a corrupted json")
}
if err := migrateV15ToV16(); err == nil {
t.Fatal("migrateConfigV15ToV16() should fail with a corrupted json")
}
if err := migrateV16ToV17(); err == nil {
t.Fatal("migrateConfigV16ToV17() should fail with a corrupted json")
}
if err := migrateV17ToV18(); err == nil {
t.Fatal("migrateConfigV17ToV18() should fail with a corrupted json")
}
if err := migrateV18ToV19(); err == nil {
t.Fatal("migrateConfigV18ToV19() should fail with a corrupted json")
}
}
// Test if all migrate code returns error with corrupted config files
func TestServerConfigMigrateCorruptedConfig(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("Init Test config failed")
}
// remove the root directory after the test ends.
defer removeAll(rootPath)
setConfigDir(rootPath)
configPath := rootPath + "/" + minioConfigFile
for i := 3; i <= 17; i++ {
// Create a corrupted config file
if err = ioutil.WriteFile(configPath, []byte(fmt.Sprintf("{ \"version\":\"%d\", \"credential\": { \"accessKey\": 1 } }", i)),
0644); err != nil {
t.Fatal("Unexpected error: ", err)
}
// Test different migrate versions and be sure they are returning an error
if err = migrateConfig(); err == nil {
t.Fatal("migrateConfig() should fail with a corrupted json")
}
}
// Create a corrupted config file for version '2'.
if err = ioutil.WriteFile(configPath, []byte("{ \"version\":\"2\", \"credentials\": { \"accessKeyId\": 1 } }"), 0644); err != nil {
t.Fatal("Unexpected error: ", err)
}
// Test different migrate versions and be sure they are returning an error
if err = migrateConfig(); err == nil {
t.Fatal("migrateConfig() should fail with a corrupted json")
}
}
+140 -258
View File
@@ -1,12 +1,22 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"os"
"path/filepath"
"sync"
"github.com/minio/minio/pkg/quick"
)
import "sync"
/////////////////// Config V1 ///////////////////
type configV1 struct {
@@ -15,28 +25,6 @@ type configV1 struct {
SecretKey string `json:"secretAccessKey"`
}
// loadConfigV1 load config
func loadConfigV1() (*configV1, error) {
configPath, err := getConfigPath()
if err != nil {
return nil, err
}
configFile := filepath.Join(configPath, "fsUsers.json")
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV1{}
c.Version = "1"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
}
/////////////////// Config V2 ///////////////////
type configV2 struct {
Version string `json:"version"`
@@ -59,29 +47,7 @@ type configV2 struct {
} `json:"fileLogger"`
}
// loadConfigV2 load config version '2'.
func loadConfigV2() (*configV2, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV2{}
c.Version = "2"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
}
/////////////////// Config V3 ///////////////////
// backendV3 type.
type backendV3 struct {
Type string `json:"type"`
@@ -133,27 +99,6 @@ type configV3 struct {
Logger loggerV3 `json:"logger"`
}
// loadConfigV3 load config version '3'.
func loadConfigV3() (*configV3, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV3{}
c.Version = "3"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
}
// logger type representing version '4' logger config.
type loggerV4 struct {
Console struct {
@@ -184,27 +129,6 @@ type configV4 struct {
Logger loggerV4 `json:"logger"`
}
// loadConfigV4 load config version '4'.
func loadConfigV4() (*configV4, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV4{}
c.Version = "4"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
}
// logger type representing version '5' logger config.
type loggerV5 struct {
Console struct {
@@ -262,31 +186,22 @@ type configV5 struct {
Logger loggerV5 `json:"logger"`
}
// loadConfigV5 load config version '5'.
func loadConfigV5() (*configV5, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV5{}
c.Version = "5"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
// consoleLogger - default logger if not other logging is enabled.
type consoleLoggerV1 struct {
Enable bool `json:"enable"`
Level string `json:"level"`
}
type fileLoggerV1 struct {
Enable bool `json:"enable"`
Filename string `json:"fileName"`
Level string `json:"level"`
}
type loggerV6 struct {
Console consoleLogger `json:"console"`
File fileLogger `json:"file"`
Syslog syslogLoggerV3 `json:"syslog"`
Console consoleLoggerV1 `json:"console"`
File fileLoggerV1 `json:"file"`
Syslog syslogLoggerV3 `json:"syslog"`
}
// configV6 server configuration version '6'.
@@ -304,27 +219,6 @@ type configV6 struct {
Notify notifierV1 `json:"notify"`
}
// loadConfigV6 load config version '6'.
func loadConfigV6() (*configV6, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &configV6{}
c.Version = "6"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
}
// Notifier represents collection of supported notification queues in version
// 1 without NATS streaming.
type notifierV1 struct {
@@ -365,27 +259,6 @@ type serverConfigV7 struct {
rwMutex *sync.RWMutex
}
// loadConfigV7 load config version '7'.
func loadConfigV7() (*serverConfigV7, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &serverConfigV7{}
c.Version = "7"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
}
// serverConfigV8 server configuration version '8'. Adds NATS notifier
// configuration.
type serverConfigV8 struct {
@@ -405,27 +278,6 @@ type serverConfigV8 struct {
rwMutex *sync.RWMutex
}
// loadConfigV8 load config version '8'.
func loadConfigV8() (*serverConfigV8, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
c := &serverConfigV8{}
c.Version = "8"
qc, err := quick.New(c)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return c, nil
}
// serverConfigV9 server configuration version '9'. Adds PostgreSQL
// notifier configuration.
type serverConfigV9 struct {
@@ -445,25 +297,10 @@ type serverConfigV9 struct {
rwMutex *sync.RWMutex
}
func loadConfigV9() (*serverConfigV9, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV9{}
srvCfg.Version = "9"
srvCfg.rwMutex = &sync.RWMutex{}
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
type loggerV7 struct {
sync.RWMutex
Console consoleLoggerV1 `json:"console"`
File fileLoggerV1 `json:"file"`
}
// serverConfigV10 server configuration version '10' which is like
@@ -477,32 +314,12 @@ type serverConfigV10 struct {
Region string `json:"region"`
// Additional error logging configuration.
Logger logger `json:"logger"`
Logger loggerV7 `json:"logger"`
// Notification queue configuration.
Notify notifierV1 `json:"notify"`
}
func loadConfigV10() (*serverConfigV10, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV10{}
srvCfg.Version = "10"
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
}
// natsNotifyV1 - structure was valid until config V 11
type natsNotifyV1 struct {
Enable bool `json:"enable"`
@@ -525,32 +342,12 @@ type serverConfigV11 struct {
Region string `json:"region"`
// Additional error logging configuration.
Logger logger `json:"logger"`
Logger loggerV7 `json:"logger"`
// Notification queue configuration.
Notify notifierV1 `json:"notify"`
}
func loadConfigV11() (*serverConfigV11, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV11{}
srvCfg.Version = "11"
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
}
// serverConfigV12 server configuration version '12' which is like
// version '11' except it adds support for NATS streaming notifications.
type serverConfigV12 struct {
@@ -561,28 +358,113 @@ type serverConfigV12 struct {
Region string `json:"region"`
// Additional error logging configuration.
Logger logger `json:"logger"`
Logger loggerV7 `json:"logger"`
// Notification queue configuration.
Notify notifierV2 `json:"notify"`
}
func loadConfigV12() (*serverConfigV12, error) {
configFile, err := getConfigFile()
if err != nil {
return nil, err
}
if _, err = os.Stat(configFile); err != nil {
return nil, err
}
srvCfg := &serverConfigV12{}
srvCfg.Version = "12"
qc, err := quick.New(srvCfg)
if err != nil {
return nil, err
}
if err := qc.Load(configFile); err != nil {
return nil, err
}
return srvCfg, nil
// serverConfigV13 server configuration version '13' which is like
// version '12' except it adds support for webhook notification.
type serverConfigV13 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
// Additional error logging configuration.
Logger *loggerV7 `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
// serverConfigV14 server configuration version '14' which is like
// version '13' except it adds support of browser param.
type serverConfigV14 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
Browser BrowserFlag `json:"browser"`
// Additional error logging configuration.
Logger *loggerV7 `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
// serverConfigV15 server configuration version '15' which is like
// version '14' except it adds mysql support
type serverConfigV15 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
Browser BrowserFlag `json:"browser"`
// Additional error logging configuration.
Logger *loggerV7 `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
// serverConfigV16 server configuration version '16' which is like
// version '15' except it makes a change to logging configuration.
type serverConfigV16 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
Browser BrowserFlag `json:"browser"`
// Additional error logging configuration.
Logger *loggers `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
// serverConfigV17 server configuration version '17' which is like
// version '16' except it adds support for "format" parameter in
// database event notification targets: PostgreSQL, MySQL, Redis and
// Elasticsearch.
type serverConfigV17 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
Browser BrowserFlag `json:"browser"`
// Additional error logging configuration.
Logger *loggers `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
// serverConfigV18 server configuration version '18' which is like
// version '17' except it adds support for "deliveryMode" parameter in
// the AMQP notification target.
type serverConfigV18 struct {
sync.RWMutex
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
Browser BrowserFlag `json:"browser"`
// Additional error logging configuration.
Logger *loggers `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
-206
View File
@@ -1,206 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"os"
"sync"
"github.com/minio/minio/pkg/quick"
)
// Read Write mutex for safe access to ServerConfig.
var serverConfigMu sync.RWMutex
// serverConfigV13 server configuration version '13' which is like
// version '12' except it adds support for webhook notification.
type serverConfigV13 struct {
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
// Additional error logging configuration.
Logger *logger `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
// newConfig - initialize a new server config, saves creds from env
// if globalIsEnvCreds is set otherwise generates a new set of keys
// and those are saved.
func newConfig(envCreds credential) error {
// Initialize server config.
srvCfg := &serverConfigV13{
Logger: &logger{},
Notify: &notifier{},
}
srvCfg.Version = globalMinioConfigVersion
srvCfg.Region = globalMinioDefaultRegion
// If env is set for a fresh start, save them to config file.
if globalIsEnvCreds {
srvCfg.SetCredential(envCreds)
} else {
srvCfg.SetCredential(newCredential())
}
// Enable console logger by default on a fresh run.
srvCfg.Logger.Console = consoleLogger{
Enable: true,
Level: "error",
}
// Make sure to initialize notification configs.
srvCfg.Notify.AMQP = make(map[string]amqpNotify)
srvCfg.Notify.AMQP["1"] = amqpNotify{}
srvCfg.Notify.ElasticSearch = make(map[string]elasticSearchNotify)
srvCfg.Notify.ElasticSearch["1"] = elasticSearchNotify{}
srvCfg.Notify.Redis = make(map[string]redisNotify)
srvCfg.Notify.Redis["1"] = redisNotify{}
srvCfg.Notify.NATS = make(map[string]natsNotify)
srvCfg.Notify.NATS["1"] = natsNotify{}
srvCfg.Notify.PostgreSQL = make(map[string]postgreSQLNotify)
srvCfg.Notify.PostgreSQL["1"] = postgreSQLNotify{}
srvCfg.Notify.Kafka = make(map[string]kafkaNotify)
srvCfg.Notify.Kafka["1"] = kafkaNotify{}
srvCfg.Notify.Webhook = make(map[string]webhookNotify)
srvCfg.Notify.Webhook["1"] = webhookNotify{}
// Create config path.
if err := createConfigPath(); err != nil {
return err
}
// hold the mutex lock before a new config is assigned.
// Save the new config globally.
// unlock the mutex.
serverConfigMu.Lock()
serverConfig = srvCfg
serverConfigMu.Unlock()
// Save config into file.
return serverConfig.Save()
}
// loadConfig - loads a new config from disk, overrides creds from env
// if globalIsEnvCreds is set otherwise serves the creds from loaded
// from the disk.
func loadConfig(envCreds credential) error {
configFile, err := getConfigFile()
if err != nil {
return err
}
if _, err = os.Stat(configFile); err != nil {
return err
}
srvCfg := &serverConfigV13{}
srvCfg.Version = globalMinioConfigVersion
qc, err := quick.New(srvCfg)
if err != nil {
return err
}
if err = qc.Load(configFile); err != nil {
return err
}
// If env is set override the credentials from config file.
if globalIsEnvCreds {
srvCfg.SetCredential(envCreds)
} else {
srvCfg.SetCredential(srvCfg.Credential)
}
// hold the mutex lock before a new config is assigned.
serverConfigMu.Lock()
// Save the loaded config globally.
serverConfig = srvCfg
serverConfigMu.Unlock()
// Set the version properly after the unmarshalled json is loaded.
serverConfig.Version = globalMinioConfigVersion
return nil
}
// serverConfig server config.
var serverConfig *serverConfigV13
// GetVersion get current config version.
func (s serverConfigV13) GetVersion() string {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Version
}
// SetRegion set new region.
func (s *serverConfigV13) SetRegion(region string) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
s.Region = region
}
// GetRegion get current region.
func (s serverConfigV13) GetRegion() string {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Region
}
// SetCredentials set new credentials.
func (s *serverConfigV13) SetCredential(creds credential) {
serverConfigMu.Lock()
defer serverConfigMu.Unlock()
// Set updated credential.
s.Credential = newCredentialWithKeys(creds.AccessKey, creds.SecretKey)
}
// GetCredentials get current credentials.
func (s serverConfigV13) GetCredential() credential {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
return s.Credential
}
// Save config.
func (s serverConfigV13) Save() error {
serverConfigMu.RLock()
defer serverConfigMu.RUnlock()
// get config file.
configFile, err := getConfigFile()
if err != nil {
return err
}
// initialize quick.
qc, err := quick.New(&s)
if err != nil {
return err
}
// Save config file.
return qc.Save(configFile)
}
-120
View File
@@ -1,120 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"reflect"
"testing"
)
func TestServerConfig(t *testing.T) {
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("Init Test config failed")
}
// remove the root directory after the test ends.
defer removeAll(rootPath)
if serverConfig.GetRegion() != globalMinioDefaultRegion {
t.Errorf("Expecting region `us-east-1` found %s", serverConfig.GetRegion())
}
// Set new region and verify.
serverConfig.SetRegion("us-west-1")
if serverConfig.GetRegion() != "us-west-1" {
t.Errorf("Expecting region `us-west-1` found %s", serverConfig.GetRegion())
}
// Set new amqp notification id.
serverConfig.Notify.SetAMQPByID("2", amqpNotify{})
savedNotifyCfg1 := serverConfig.Notify.GetAMQPByID("2")
if !reflect.DeepEqual(savedNotifyCfg1, amqpNotify{}) {
t.Errorf("Expecting AMQP config %#v found %#v", amqpNotify{}, savedNotifyCfg1)
}
// Set new elastic search notification id.
serverConfig.Notify.SetElasticSearchByID("2", elasticSearchNotify{})
savedNotifyCfg2 := serverConfig.Notify.GetElasticSearchByID("2")
if !reflect.DeepEqual(savedNotifyCfg2, elasticSearchNotify{}) {
t.Errorf("Expecting Elasticsearch config %#v found %#v", elasticSearchNotify{}, savedNotifyCfg2)
}
// Set new redis notification id.
serverConfig.Notify.SetRedisByID("2", redisNotify{})
savedNotifyCfg3 := serverConfig.Notify.GetRedisByID("2")
if !reflect.DeepEqual(savedNotifyCfg3, redisNotify{}) {
t.Errorf("Expecting Redis config %#v found %#v", redisNotify{}, savedNotifyCfg3)
}
// Set new kafka notification id.
serverConfig.Notify.SetKafkaByID("2", kafkaNotify{})
savedNotifyCfg4 := serverConfig.Notify.GetKafkaByID("2")
if !reflect.DeepEqual(savedNotifyCfg4, kafkaNotify{}) {
t.Errorf("Expecting Kafka config %#v found %#v", kafkaNotify{}, savedNotifyCfg4)
}
// Set new Webhook notification id.
serverConfig.Notify.SetWebhookByID("2", webhookNotify{})
savedNotifyCfg5 := serverConfig.Notify.GetWebhookByID("2")
if !reflect.DeepEqual(savedNotifyCfg5, webhookNotify{}) {
t.Errorf("Expecting Webhook config %#v found %#v", webhookNotify{}, savedNotifyCfg3)
}
// Set new console logger.
serverConfig.Logger.SetConsole(consoleLogger{
Enable: true,
})
consoleCfg := serverConfig.Logger.GetConsole()
if !reflect.DeepEqual(consoleCfg, consoleLogger{Enable: true}) {
t.Errorf("Expecting console logger config %#v found %#v", consoleLogger{Enable: true}, consoleCfg)
}
// Set new console logger.
serverConfig.Logger.SetConsole(consoleLogger{
Enable: false,
})
// Set new file logger.
serverConfig.Logger.SetFile(fileLogger{
Enable: true,
})
fileCfg := serverConfig.Logger.GetFile()
if !reflect.DeepEqual(fileCfg, fileLogger{Enable: true}) {
t.Errorf("Expecting file logger config %#v found %#v", fileLogger{Enable: true}, consoleCfg)
}
// Set new file logger.
serverConfig.Logger.SetFile(fileLogger{
Enable: false,
})
// Match version.
if serverConfig.GetVersion() != globalMinioConfigVersion {
t.Errorf("Expecting version %s found %s", serverConfig.GetVersion(), globalMinioConfigVersion)
}
// Attempt to save.
if err := serverConfig.Save(); err != nil {
t.Fatalf("Unable to save updated config file %s", err)
}
// Do this only once here.
setGlobalConfigPath(rootPath)
// Initialize server config.
if err := loadConfig(credential{}); err != nil {
t.Fatalf("Unable to initialize from updated config file %s", err)
}
}
+322
View File
@@ -0,0 +1,322 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"errors"
"fmt"
"io/ioutil"
"sync"
"github.com/minio/minio/pkg/quick"
"github.com/tidwall/gjson"
)
// Config version
const v19 = "19"
var (
// serverConfig server config.
serverConfig *serverConfigV19
serverConfigMu sync.RWMutex
)
// serverConfigV19 server configuration version '19' which is like
// version '18' except it adds support for MQTT notifications.
type serverConfigV19 struct {
sync.RWMutex
Version string `json:"version"`
// S3 API configuration.
Credential credential `json:"credential"`
Region string `json:"region"`
Browser BrowserFlag `json:"browser"`
// Additional error logging configuration.
Logger *loggers `json:"logger"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
}
// GetVersion get current config version.
func (s *serverConfigV19) GetVersion() string {
s.RLock()
defer s.RUnlock()
return s.Version
}
// SetRegion set new region.
func (s *serverConfigV19) SetRegion(region string) {
s.Lock()
defer s.Unlock()
s.Region = region
}
// GetRegion get current region.
func (s *serverConfigV19) GetRegion() string {
s.RLock()
defer s.RUnlock()
return s.Region
}
// SetCredentials set new credentials.
func (s *serverConfigV19) SetCredential(creds credential) {
s.Lock()
defer s.Unlock()
// Set updated credential.
s.Credential = creds
}
// GetCredentials get current credentials.
func (s *serverConfigV19) GetCredential() credential {
s.RLock()
defer s.RUnlock()
return s.Credential
}
// SetBrowser set if browser is enabled.
func (s *serverConfigV19) SetBrowser(b bool) {
s.Lock()
defer s.Unlock()
// Set the new value.
s.Browser = BrowserFlag(b)
}
// GetCredentials get current credentials.
func (s *serverConfigV19) GetBrowser() bool {
s.RLock()
defer s.RUnlock()
return bool(s.Browser)
}
// Save config.
func (s *serverConfigV19) Save() error {
s.RLock()
defer s.RUnlock()
// Save config file.
return quick.Save(getConfigFile(), s)
}
func newServerConfigV19() *serverConfigV19 {
srvCfg := &serverConfigV19{
Version: v19,
Credential: mustGetNewCredential(),
Region: globalMinioDefaultRegion,
Browser: true,
Logger: &loggers{},
Notify: &notifier{},
}
// Enable console logger by default on a fresh run.
srvCfg.Logger.Console = NewConsoleLogger()
// Make sure to initialize notification configs.
srvCfg.Notify.AMQP = make(map[string]amqpNotify)
srvCfg.Notify.AMQP["1"] = amqpNotify{}
srvCfg.Notify.MQTT = make(map[string]mqttNotify)
srvCfg.Notify.MQTT["1"] = mqttNotify{}
srvCfg.Notify.ElasticSearch = make(map[string]elasticSearchNotify)
srvCfg.Notify.ElasticSearch["1"] = elasticSearchNotify{}
srvCfg.Notify.Redis = make(map[string]redisNotify)
srvCfg.Notify.Redis["1"] = redisNotify{}
srvCfg.Notify.NATS = make(map[string]natsNotify)
srvCfg.Notify.NATS["1"] = natsNotify{}
srvCfg.Notify.PostgreSQL = make(map[string]postgreSQLNotify)
srvCfg.Notify.PostgreSQL["1"] = postgreSQLNotify{}
srvCfg.Notify.MySQL = make(map[string]mySQLNotify)
srvCfg.Notify.MySQL["1"] = mySQLNotify{}
srvCfg.Notify.Kafka = make(map[string]kafkaNotify)
srvCfg.Notify.Kafka["1"] = kafkaNotify{}
srvCfg.Notify.Webhook = make(map[string]webhookNotify)
srvCfg.Notify.Webhook["1"] = webhookNotify{}
return srvCfg
}
// newConfig - initialize a new server config, saves env parameters if
// found, otherwise use default parameters
func newConfig() error {
// Initialize server config.
srvCfg := newServerConfigV19()
// If env is set override the credentials from config file.
if globalIsEnvCreds {
srvCfg.SetCredential(globalActiveCred)
}
if globalIsEnvBrowser {
srvCfg.SetBrowser(globalIsBrowserEnabled)
}
if globalIsEnvRegion {
srvCfg.SetRegion(globalServerRegion)
}
// hold the mutex lock before a new config is assigned.
// Save the new config globally.
// unlock the mutex.
serverConfigMu.Lock()
serverConfig = srvCfg
serverConfigMu.Unlock()
// Save config into file.
return serverConfig.Save()
}
// doCheckDupJSONKeys recursively detects duplicate json keys
func doCheckDupJSONKeys(key, value gjson.Result) error {
// Key occurrences map of the current scope to count
// if there is any duplicated json key.
keysOcc := make(map[string]int)
// Holds the found error
var checkErr error
// Iterate over keys in the current json scope
value.ForEach(func(k, v gjson.Result) bool {
// If current key is not null, check if its
// value contains some duplicated keys.
if k.Type != gjson.Null {
keysOcc[k.String()]++
checkErr = doCheckDupJSONKeys(k, v)
}
return checkErr == nil
})
// Check found err
if checkErr != nil {
return errors.New(key.String() + " => " + checkErr.Error())
}
// Check for duplicated keys
for k, v := range keysOcc {
if v > 1 {
return errors.New(key.String() + " => `" + k + "` entry is duplicated")
}
}
return nil
}
// Check recursively if a key is duplicated in the same json scope
// e.g.:
// `{ "key" : { "key" ..` is accepted
// `{ "key" : { "subkey" : "val1", "subkey": "val2" ..` throws subkey duplicated error
func checkDupJSONKeys(json string) error {
// Parse config with gjson library
config := gjson.Parse(json)
// Create a fake rootKey since root json doesn't seem to have representation
// in gjson library.
rootKey := gjson.Result{Type: gjson.String, Str: minioConfigFile}
// Check if loaded json contains any duplicated keys
return doCheckDupJSONKeys(rootKey, config)
}
// getValidConfig - returns valid server configuration
func getValidConfig() (*serverConfigV19, error) {
srvCfg := &serverConfigV19{
Region: globalMinioDefaultRegion,
Browser: true,
}
configFile := getConfigFile()
if _, err := quick.Load(configFile, srvCfg); err != nil {
return nil, err
}
if srvCfg.Version != v19 {
return nil, fmt.Errorf("configuration version mismatch. Expected: %s, Got: %s", v19, srvCfg.Version)
}
// Load config file json and check for duplication json keys
jsonBytes, err := ioutil.ReadFile(configFile)
if err != nil {
return nil, err
}
if err = checkDupJSONKeys(string(jsonBytes)); err != nil {
return nil, err
}
// Validate credential fields only when
// they are not set via the environment
// Error out if global is env credential is not set and config has invalid credential
if !globalIsEnvCreds && !srvCfg.Credential.IsValid() {
return nil, errors.New("invalid credential in config file " + configFile)
}
// Validate logger field
if err = srvCfg.Logger.Validate(); err != nil {
return nil, err
}
// Validate notify field
if err = srvCfg.Notify.Validate(); err != nil {
return nil, err
}
return srvCfg, nil
}
// loadConfig - loads a new config from disk, overrides params from env
// if found and valid
func loadConfig() error {
srvCfg, err := getValidConfig()
if err != nil {
return err
}
// If env is set override the credentials from config file.
if globalIsEnvCreds {
srvCfg.SetCredential(globalActiveCred)
}
if globalIsEnvBrowser {
srvCfg.SetBrowser(globalIsBrowserEnabled)
}
if globalIsEnvRegion {
srvCfg.SetRegion(globalServerRegion)
}
// hold the mutex lock before a new config is assigned.
serverConfigMu.Lock()
serverConfig = srvCfg
if !globalIsEnvCreds {
globalActiveCred = serverConfig.GetCredential()
}
if !globalIsEnvBrowser {
globalIsBrowserEnabled = serverConfig.GetBrowser()
}
if !globalIsEnvRegion {
globalServerRegion = serverConfig.GetRegion()
}
serverConfigMu.Unlock()
return nil
}

Some files were not shown because too many files have changed in this diff Show More