Compare commits

..

78 Commits

Author SHA1 Message Date
dingjs 289457568c Translate #5365 to Chinese. (#5439) 2018-02-08 17:00:03 -08:00
Harshavardhana fd3897d0c3 Move to go1.9.4 with recent security release (#5502) 2018-02-08 14:33:22 +05:30
Krishna Srinivas 047b7aff0c Seek to offset 0 after Truncate() (#5375) 2018-02-06 15:37:48 -08:00
Harshavardhana 1164fc60f3 Bring semantic versioning to provide for rolling upgrades (#5495)
This PR brings semver capabilities in our RPC layer to
ensure that we can upgrade the servers in rolling fashion
while keeping I/O in progress. This is only a framework change
the functionality remains the same as such and we do not
have any special API changes for now. But in future when
we bring in API changes we will be able to upgrade servers
without a downtime.

Additional change in this PR is to not abort when serverVersions
mismatch in a distributed cluster, instead wait for the quorum
treat the situation as if the server is down. This allows
for administrator to properly upgrade all the servers in the cluster.

Fixes #5393
2018-02-06 15:07:17 -08:00
kannappanr 48218272cc Document object name limitations on Windows (#5491)
Fixes #5161
2018-02-03 19:57:40 +05:30
Harshavardhana 0c880bb852 Deprecate and remove in-memory object caching (#5481)
in-memory caching cannot be cleanly implemented
without the access to GC which Go doesn't naturally
provide. At times we have seen that object caching
is more of an hindrance rather than a boon for
our use cases.

Removing it completely from our implementation
  related to #5160 and #5182
2018-02-02 10:17:13 -08:00
Harshavardhana 1ebbc2ce88 Make sure to convert the disk errors to object errors (#5480)
Fixes a bug introduced in the directory support PR, with
this fix s3fs works properly.
2018-02-02 14:04:15 +05:30
A. Elleuch da2faa19a1 Reduce Minio access key minimum length to 3 (#5478)
This is a generic minimum value. The current reason is to support
Azure blob storage accounts name whose length is less than 5. 3 is the
minimum length for Azure.
2018-02-02 09:13:30 +05:30
Krishna Srinivas 2afd196c83 Quorum based listing for XL (#5475)
fixes #5380
2018-02-01 10:47:49 -08:00
Krishna Srinivas b606ba3f81 fs.json file should be closed in CompleteMultipartUpload (#5482) 2018-02-01 15:27:12 +05:30
Harshavardhana 3316dbc037 simplify storage class validation (#5470)
Check if the storage class is set in an
non XL setup instead of relying on `globalEndpoints`
value. Also converge the checks for both SS
and RRS parity configuration.

This PR also removes redundant `tt.name` in all
test cases, since each testcase doesn't need to
be numbered explicitly they are numbered implicitly.
2018-02-01 13:00:07 +05:30
Harshavardhana 033cfb5cef Remove stale code from minio server (#5479) 2018-01-31 18:28:28 -08:00
Krishna Srinivas 3b2486ebaf Lock free multipart backend implementation for FS (#5401) 2018-01-31 13:17:24 -08:00
Aditya Manthramurthy 018813b98f Fix configuration handling bugs: (#5473)
* Update the GetConfig admin API to use the latest version of
  configuration, along with fixes to the corresponding RPCs.
* Remove mutex inside the configuration struct, and inside
  notification struct.
* Use global config mutex where needed.
* Add `serverConfig.ConfigDiff()` that provides a more granular diff
  of what is different between two configurations.
2018-01-31 08:15:54 -08:00
ebozduman e608e05cda Removes capitalization of error causes (#5468) 2018-01-30 21:42:15 -08:00
Harshavardhana 3ea28e9771 Support creating directories on erasure coded backend (#5443)
This PR continues from #5049 where we started supporting
directories for erasure coded backend
2018-01-30 08:13:13 +05:30
Krishna Srinivas 45c35b3544 Autocorrect user provided Azure endpoint (#5417)
fixes #5373
2018-01-29 10:30:08 -08:00
Andreas Auernhammer 09a9002f12 add documentation about PKCS-8 encrypted RSA keys (#5454)
This change adds documentation about PKCS-8 vs PKCS-1 pitfalls. It 
also provides a command to convert encrypted PKCS-8 RSA keys to 
encrypted PKCS-1 RSA keys.

Fixes #5453
2018-01-27 09:30:02 +05:30
Aditya Manthramurthy 5cdcc73bd5 Admin API auth and heal related fixes (#5445)
- Fetch region for auth from global state
- Fix SHA256 handling for empty body in heal API
2018-01-25 19:24:00 +05:30
poornas 2dd117f647 fix testcases to init nslock properly (#5429) 2018-01-24 09:04:09 +05:30
Harshavardhana 2d19663fef Update dockerfile go version to 1.9.2 (#5441) 2018-01-23 17:19:19 +05:30
Aditya Manthramurthy 254b05e314 Fix locking in some admin APIs: (#5438)
- read lock for get config
- write lock for update creds
- write lock for format file
2018-01-22 18:09:12 -08:00
Aditya Manthramurthy a003de72bf Update madmin doc (fixes #5432) (#5433) 2018-01-22 16:10:43 -08:00
Aditya Manthramurthy a337ea4d11 Move admin APIs to new path and add redesigned heal APIs (#5351)
- Changes related to moving admin APIs
   - admin APIs now have an endpoint under /minio/admin
   - admin APIs are now versioned - a new API to server the version is
     added at "GET /minio/admin/version" and all API operations have the
     path prefix /minio/admin/v1/<operation>
   - new service stop API added
   - credentials change API is moved to /minio/admin/v1/config/credential
   - credentials change API and configuration get/set API now require TLS
     so that credentials are protected
   - all API requests now receive JSON
   - heal APIs are disabled as they will be changed substantially

- Heal API changes
   Heal API is now provided at a single endpoint with the ability for a
   client to start a heal sequence on all the data in the server, a
   single bucket, or under a prefix within a bucket.

   When a heal sequence is started, the server returns a unique token
   that needs to be used for subsequent 'status' requests to fetch heal
   results.

   On each status request from the client, the server returns heal result
   records that it has accumulated since the previous status request. The
   server accumulates upto 1000 records and pauses healing further
   objects until the client requests for status. If the client does not
   request any further records for a long time, the server aborts the
   heal sequence automatically.

   A heal result record is returned for each entity healed on the server,
   such as system metadata, object metadata, buckets and objects, and has
   information about the before and after states on each disk.

   A client may request to force restart a heal sequence - this causes
   the running heal sequence to be aborted at the next safe spot and
   starts a new heal sequence.
2018-01-22 14:54:55 -08:00
Harshavardhana f3f09ed14e Fix a bug in dsync initialization and communication (#5428)
In current implementation we used as many dsync clients
as per number of endpoints(along with path) which is not
the expected implementation. The implementation of Dsync
was expected to be just for the endpoint Host alone such
that if you have 4 servers and each with 4 disks we need
to only have 4 dsync clients and 4 dsync servers. But
we currently had 8 clients, servers which in-fact is
unexpected and should be avoided.

This PR brings the implementation back to its original
intention. This issue was found #5160
2018-01-22 10:25:10 -08:00
Harshavardhana bb73c84b10 Add notification structure link (#5426)
Fixes #4545
2018-01-20 09:23:09 +05:30
Harshavardhana e19eddd759 Remove requirement for custom RPCClient (#5405)
This change is a simplification over existing
code since it is not required to have a separate
RPCClient structure instead keep authRPCClient can
do the same job.

There is no code which directly uses netRPCClient(),
keeping authRPCClient is better and simpler. This
simplication also allows for removal of multiple
levels of locking code per object.

Observed in #5160
2018-01-19 16:38:47 -08:00
Andreas Auernhammer 7f99cc9768 add HighwayHash256 support (#5359)
This change adds the HighwayHash256 PRF as bitrot protection / detection
algorithm. Since HighwayHash256 requires a 256 bit we generate a random
key from the first 100 decimals of π - See nothing-up-my-sleeve-numbers.
This key is fixed forever and tied to the HighwayHash256 bitrot algorithm.

Fixes #5358
2018-01-19 10:18:21 -08:00
Aditya Manthramurthy 2760409656 Remove dead code and associated dead code warning (#5424) 2018-01-19 10:16:21 -08:00
fossabot 1f13235cbd Add license scan report and status (#5430) 2018-01-19 13:16:59 +05:30
poornas dd5a3289dd fix: listobjects return empty response for invalid prefix/marker (#5425)
Currently minio server returns a NotImplemented error when marker
is not common with prefix. Instead, return an empty ListObjectsResponse
2018-01-18 14:39:39 -08:00
Harshavardhana b6e4f053a3 Fix lock rpc server maintenance loop go-routine leak (#5423)
The problem was after the globalServiceDoneCh receives a
message, we cleanly stop the ticker as expected. But the
go-routine where the `select` loop is running is never
returned from. The stage at which point this may occur
i.e server is being restarted, doesn't seriously affect
servers usage. But any build up like this on server has
consequences as the new functionality would come in future.
2018-01-18 14:39:24 -08:00
Minio Trusted 1c3f55ff64 Update yaml files to latest version RELEASE.2018-01-18T20-33-21Z 2018-01-18 20:41:53 +00:00
Nitish Tiwari e2d5a87b26 Fix free and total space reported in startup banner (#5419)
With storage class support, the free and total space
reported in Minio XL startup banner should be based on
totalDisks - standardClassParityDisks, instead of totalDisks/2.

fixes #5416
2018-01-17 11:25:51 -08:00
Andreas Auernhammer d0a43af616 replace all "crypto/sha256" with "github.com/minio/sha256-simd" (#5391)
This change replaces all imports of "crypto/sha256" with
"github.com/minio/sha256-simd". The sha256-simd package
is faster on ARM64 (NEON instructions) and can take advantage
of AVX-512 in certain scenarios.

Fixes #5374
2018-01-17 10:54:31 -08:00
Paul Stack a020a70484 gateway/manta: Bump manta dependencies (#5414)
Internally, triton-go, what manta minio is built on, changed it's internal
error handling. This means we no longer need to unwrap specific error types

This doesn't change any manta minio functionality - it just changes how errors are
handled internally and adds a wrapper for a 404 error
2018-01-17 10:38:39 -08:00
Andreas Auernhammer 3f09c17bfe fix authentication bypass against Admin-API (#5412)
This change fixes an authentication bypass attack against the
minio Admin-API. Therefore the Admin-API rejects now all types of
requests except valid signature V2 and signature V4 requests - this
includes signature V2/V4 pre-signed requests.

Fixes #5411
2018-01-17 10:36:25 -08:00
ebozduman 24d9d7e5fa Removes logrus package and refactors logging messages (#5293)
This fix removes logrus package dependency and refactors the console
logging as the only logging mechanism by removing file logging support.
It rearranges the log message format and adds stack trace information
whenever trace information is not available in the error structure.
It also adds `--json` flag support for server logging.
When minio server is started with `--json` flag, all log messages are
displayed in json format, with no start-up and informational log
messages.
Fixes #5265 #5220 #5197
2018-01-17 07:24:46 -08:00
kannappanr 85580fe0d6 Document pre-existing data in fs mode (#5365)
Document that pre-existing data in fs mode is safe and 
accessible

Fixes #5050
2018-01-17 20:44:31 +05:30
Nitish Tiwari 8a1dc10c60 Update storage class related documents (#5399)
- Add storage usage info in storage class doc
- Update distributed & erasure code doc with info on storage class
2018-01-17 14:52:42 +05:30
Krishnan Parthasarathi 17301fe45d Don't delete lock ops entry during state change (#5388)
lock ops entry is removed in deleteLockEntryForOps, it shouldn't be removed
in status*To* functions.
2018-01-16 12:00:12 -08:00
Aditya Manthramurthy aa7e5c71e9 Remove upload healing related dead code (#5404) 2018-01-15 18:20:39 -08:00
Kaan Kabalak 78a641fc6a Fix multi-file dropzone upload issue causing bucket listing duplication (#5392)
This commit fixes the order of the functions inside the selectPrefix
function due to the fact that, as multiple files were being uploaded,
the resetObjects function (that clears the object list) ran repeatedly
for each of these objects, right before the appendObjects function (that
appends the objects being uploaded to the current list of objects) also
ran for all of these objects. This caused all the objects in the bucket
to be repeated in the list for the number of objects that were dragged
into the dropzone.
2018-01-13 22:45:20 +05:30
Harshavardhana 12f67d47f1 Fix a possible race during PutObject() (#5376)
Under any concurrent removeObjects in progress
might have removed the parents of the same prefix
for which there is an ongoing putObject request.
An inconsistent situation may arise as explained
below even under sufficient locking.

PutObject is almost successful at the last stage when
a temporary file is renamed to its actual namespace
at `a/b/c/object1`. Concurrently a RemoveObject is
also in progress at the same prefix for an `a/b/c/object2`.

To create the object1 at location `a/b/c` PutObject has
to create all the parents recursively.

```
a/b/c - os.MkdirAll loops through has now created
        'a/' and 'b/' about to create 'c/'
a/b/c/object2 - at this point 'c/' and 'object2'
        are deleted about to delete b/
```

Now for os.MkdirAll loop the expected situation is
that top level parent 'a/b/' exists which it created
, such that it can create 'c/' - since removeObject
and putObject do not compete for lock due to holding
locks at different resources. removeObject proceeds
to delete parent 'b/' since 'c/' is not yet present,
once deleted 'os.MkdirAll' would receive an error as
syscall.ENOENT which would fail the putObject request.

This PR tries to address this issue by implementing
a safer/guarded approach where we would retry an operation
such as `os.MkdirAll` and `os.Rename` if both operations
observe syscall.ENOENT.

Fixes #5254
2018-01-13 22:43:02 +05:30
poornas 0bb6247056 Move nslocking from s3 layer to object layer (#5382)
Fixes #5350
2018-01-13 10:04:52 +05:30
Andreas Auernhammer dd202a1a5f restrict TLS cipher suites of the server (#5245)
This change restircts the supported cipher suites of the minio server.
The server only supports AEAD ciphers (Chacha20Poly1305 and 
AES-GCM)

The supported cipher suites are:
 - tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
 - tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
 - tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 - tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
 - tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 - tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Fixes #5244 and #5291
2018-01-13 09:12:11 +05:30
Nitish Tiwari ede504400f Add validation of xlMeta ErasureInfo field (#5389) 2018-01-12 18:16:30 +05:30
Harshavardhana 4b2d04c86f Add chroot environment doc for minio (#5366)
Fixes #4659
2018-01-12 07:55:40 +05:30
Nitish Tiwari 42633748db Update madmin package to return storage class parity (#5387)
After the addition of Storage Class support, readQuorum
and writeQuorum are decided on a per object basis, instead
of deployment wide static quorums.

This PR updates madmin api to remove readQuorum/writeQuorum
and add Standard storage class and reduced redundancy storage
class parity as return values. Since these parity values are
used to decide the quorum for each object.

Fixes #5378
2018-01-12 07:52:52 +05:30
Aditya Manthramurthy cd22feecf8 Remove healing of incomplete multipart uploads (#5390)
Since the server performs automatic clean-up of multipart uploads that
have not been resumed for more than a couple of weeks, it was decided
to remove functionality to heal multipart uploads.
2018-01-11 15:07:43 -08:00
kannappanr 20584dc08f Remove unnecessary errors printed on the console (#5386)
Some of the errors printed on server console can be
removed as those error message is unnecessary.

Fixes #5385
2018-01-11 11:42:05 -08:00
Aditya Manthramurthy 8e4eb591c1 Update error response when heal is not implemented (#5383) 2018-01-11 10:21:41 -08:00
Nitish Tiwari 1b721d76b1 Assume standard storage class if not set in metadata (#5370)
If STANDARD storage class is set before starting up Minio server, 
but x-amz-storage-class metadata field is not set in a PutObject 
request, Minio server defaults to N/2 data and N/2 parity disks.

This PR changes the behaviour to use data and parity disks set in
STANDARD storage class, even if x-amz-storage-class metadata 
field is not present in PutObject requests.
2018-01-11 14:58:12 +05:30
Aditya Manthramurthy f413224b24 Fix config set handler (#5384)
- Return error when the config JSON has duplicate keys (fixes #5286)

- Limit size of configuration file provided to 256KiB - this prevents
  another form of DoS
2018-01-11 12:36:36 +05:30
Harshavardhana b526cd7e55 Remove requirement for issued at JWT claims (#5364)
Remove the requirement for IssuedAt claims from JWT
for now, since we do not currently have a way to provide
a leeway window for validating the claims. Expiry does
the same checks as IssuedAt with an expiry window.

We do not need it right now since we have clock skew check
in our RPC layer to handle this correctly.

rpc-common.go
```
func isRequestTimeAllowed(requestTime time.Time) bool {
        // Check whether request time is within acceptable skew time.
        utcNow := UTCNow()
        return !(requestTime.Sub(utcNow) > rpcSkewTimeAllowed ||
                utcNow.Sub(requestTime) > rpcSkewTimeAllowed)
}
```

Once the PR upstream is merged https://github.com/dgrijalva/jwt-go/pull/139
We can bring in support for leeway later.

Fixes #5237
2018-01-10 10:34:00 -08:00
Aditya Manthramurthy 3f8379d07d Update Elasticsearch documentation with authentication information (#5381)
- Add documentation to show how to supply credential to access a
  secured elasticsearch server.

Fixes #5329
2018-01-10 09:20:42 +05:30
Harshavardhana 7350543f24 Allow x-amz-content-sha256 to be optional for PutObject() (#5340)
x-amz-content-sha256 can be optional for any AWS signature v4
requests, make sure to skip sha256 calculation when payload
checksum is not set.

Here is the overall expected behavior

** Signed request **
- X-Amz-Content-Sha256 is set to 'empty' or some 'value'  or its
  not 'UNSIGNED-PAYLOAD'- use it to validate the incoming payload.
- X-Amz-Content-Sha256 is set to 'UNSIGNED-PAYLOAD' - skip checksum verification
- X-Amz-Content-Sha256 is not set we use emptySHA256

** Presigned request **
- X-Amz-Content-Sha256 is set to 'empty' or some 'value'  or its
  not 'UNSIGNED-PAYLOAD'- use it to validate the incoming payload
- X-Amz-Content-Sha256 is set to 'UNSIGNED-PAYLOAD' - skip checksum verification
- X-Amz-Content-Sha256 is not set we use 'UNSIGNED-PAYLOAD'

Fixes #5339
2018-01-09 12:49:50 +05:30
Nitish Tiwari 56bde5df31 Refactor storage class parsing for Gateway mode (#5331)
This PR updates the behaviour to print relevant error message
if storage class is set in config.json for gateway

This PR also fixes the case where storage class set via
environment variables is not parsed properly into config.json.
2018-01-08 22:26:13 -08:00
Paul Trunk bd9cdcf379 Add custom secret names for Docker (#5355) 2018-01-09 10:46:25 +05:30
Krishna Srinivas 7c72d14027 Separate the codebase for XL and FS format.json related code (#5317) 2018-01-08 14:30:55 -08:00
Harshavardhana ccd9767b7a Remove incomplete older chinese translation doc (#5368) 2018-01-08 12:48:59 +05:30
Harshavardhana dae8193bd4 Remove duplicate http constants (#5367) 2018-01-08 10:17:48 +05:30
kannappanr 1de3bd6911 Save http trace to a file (#5300)
Save http trace to a file instead of displaying it onto the console.
the environment variable MINIO_HTTP_TRACE will be a filepath instead
of a boolean.

This to handle the scenario where both json and http tracing are
turned on. In that case, both http trace and json output are displayed
on the screen making the json not parsable. Loging this trace onto
a file helps us avoid that scenario.

Fixes #5263
2018-01-05 11:24:31 -08:00
Kaan Kabalak de2ce5acb4 Correct color code of Excel file icon (#5361)
Fixes #5352
2018-01-05 16:19:41 +05:30
Paul Stack a1a98617ca gateway/manta: Add support for RBAC (#5332)
Manta has the ability to allow users to authenticate with a 
username other than the main account. We want to expose 
this functionality to minio manta gateway.
2018-01-05 13:30:29 +05:30
Andreas Auernhammer b85c75996d add support for encrypted TLS private keys (#5308)
This change adds support for password-protected private keys.
If the private key is encrypted the server tries to decrypt
the key with the password provided by the env variable 
MINIO_CERT_PASSWD.

Fixes #5302
2018-01-05 13:18:08 +05:30
Harshavardhana cc2497f52f Exitcode with '1' when update is available (#5354)
--quiet should simply update the binary without any prompt.

Fixes #5347
2018-01-04 21:26:43 +05:30
Nitish Tiwari 1e5fb4b79a Fix storage class related issues (#5338)
- Update startup banner to print storage class in capitals. This
makes it easier to identify different storage classes available.

- Update response metadata to not send STANDARD storage class.
This is in accordance with AWS S3 behaviour.

- Update minio-go library to bring in storage class related
changes. This is needed to make transparent translation of
storage class headers for Minio S3 Gateway.
2018-01-04 11:44:45 +05:30
kannappanr 6f7c6fc560 Honor browser enabled config value in startup message (#5313)
Currently, browser access information is displayed without checking
if browser enabled flag is turned off in config.json. Fixing it to
hide the information if the flag is turned off.

Fixes #5312
2018-01-04 11:00:52 +05:30
Harshavardhana c0721164be Automatically set goroutines based on shardSize (#5346)
Update reedsolomon library to enable feature to automatically
set number of go-routines based on the input shard size,
since shard size is sort of a constant in Minio for
objects > 10MiB (default blocksize)

klauspost reported around 15-20% improvement in performance
numbers on older systems such as AVX and SSE3

```
name                  old speed      new speed      delta
Encode10x2x10000-8    5.45GB/s ± 1%  6.22GB/s ± 1%  +14.20%    (p=0.000 n=9+9)
Encode100x20x10000-8  1.44GB/s ± 1%  1.64GB/s ± 1%  +13.77%  (p=0.000 n=10+10)
Encode17x3x1M-8       10.0GB/s ± 5%  12.0GB/s ± 1%  +19.88%  (p=0.000 n=10+10)
Encode10x4x16M-8      7.81GB/s ± 5%  8.56GB/s ± 5%   +9.58%   (p=0.000 n=10+9)
Encode5x2x1M-8        15.3GB/s ± 2%  19.6GB/s ± 2%  +28.57%   (p=0.000 n=9+10)
Encode10x2x1M-8       12.2GB/s ± 5%  15.0GB/s ± 5%  +22.45%  (p=0.000 n=10+10)
Encode10x4x1M-8       7.84GB/s ± 1%  9.03GB/s ± 1%  +15.19%    (p=0.000 n=9+9)
Encode50x20x1M-8      1.73GB/s ± 4%  2.09GB/s ± 4%  +20.59%   (p=0.000 n=10+9)
Encode17x3x16M-8      10.6GB/s ± 1%  11.7GB/s ± 4%  +10.12%   (p=0.000 n=8+10)
```
2018-01-03 13:47:22 -08:00
Minio Trusted b1fb550d5c Update yaml files to latest version RELEASE.2018-01-02T23-07-00Z 2018-01-02 23:11:15 +00:00
Andreas Auernhammer a6318dbdaf fix timing oracle attack against signature V2/V4 verification (#5335)
This change replaces the non-constant time comparison of
request signatures with a constant time implementation. This
prevents a timing attack which can be used to learn a valid 
signature for a request without knowing the secret key.

Fixes #5334
2018-01-02 12:00:02 +05:30
Harshavardhana e39d7ddb0f Fix PostPolicy form tests without hardcoded dates (#5337)
Fixes #5336
2018-01-01 07:28:10 +05:30
Kaan Kabalak 659f724f4c Integrate existing remove bucket functionality from newux to current UI (#5289)
This commit takes the existing remove bucket functionality written by
brendanashworth, integrates it to the current UI with a dropdown for
each bucket, and fixes small issues that were present, like the dropdown
not disappearing after the user clicks on 'Delete' for certain buckets.
This feature only deletes a bucket that is empty (that has no objects).

Fixes #4166
2017-12-29 18:45:44 +05:30
Nitish Tiwari baaf67d82e Update config.json guide with details of version 22 (#5328)
Fixes #5296
2017-12-28 23:04:44 +05:30
A. Elleuch 2244adff07 fix: Better printing of XL config init error (#5284) 2017-12-28 23:02:48 +05:30
Nitish Tiwari e3d841ffd1 Fix config.json parsing to fetch correct storage class (#5327) 2017-12-28 14:19:45 +05:30
Minio Trusted 751632d79e Update yaml files to lastest version RELEASE.2017-12-28T01-21-00Z 2017-12-28 01:26:09 +00:00
295 changed files with 17544 additions and 9612 deletions
+1 -1
View File
@@ -26,4 +26,4 @@ after_success:
- bash <(curl -s https://codecov.io/bash)
go:
- 1.9.1
- 1.9.4
+3 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.9.1-alpine3.6
FROM golang:1.9.4-alpine3.6
MAINTAINER Minio Inc <dev@minio.io>
@@ -6,6 +6,8 @@ ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
WORKDIR /go/src/github.com/minio/
+3 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.9.1-alpine3.6
FROM golang:1.9.4-alpine3.6
MAINTAINER Minio Inc <dev@minio.io>
@@ -6,6 +6,8 @@ ENV GOPATH /go
ENV PATH $PATH:$GOPATH/bin
ENV CGO_ENABLED 0
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
WORKDIR /go/src/github.com/minio/
+2
View File
@@ -5,6 +5,8 @@ MAINTAINER Minio Inc <dev@minio.io>
COPY dockerscripts/docker-entrypoint.sh dockerscripts/healthcheck.sh /usr/bin/
ENV MINIO_UPDATE off
ENV MINIO_ACCESS_KEY_FILE=access_key \
MINIO_SECRET_KEY_FILE=secret_key
RUN \
apk add --no-cache ca-certificates && \
+10
View File
@@ -1,5 +1,6 @@
# Minio Quickstart Guide
[![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/) [![codecov](https://codecov.io/gh/minio/minio/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/minio) [![Snap Status](https://build.snapcraft.io/badge/minio/minio.svg)](https://build.snapcraft.io/user/minio/minio)
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fminio%2Fminio.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fminio%2Fminio?ref=badge_shield)
Minio is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
@@ -99,6 +100,11 @@ Minio Server comes with an embedded web based object browser. Point your web bro
## Test using Minio Client `mc`
`mc` provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. Follow the Minio Client [Quickstart Guide](https://docs.minio.io/docs/minio-client-quickstart-guide) for further instructions.
## Pre-existing data
When deployed on a single drive, Minio server lets clients access any pre-existing data in the data directory. For example, if Minio is started with the command `minio server /mnt/data`, any pre-existing data in the `/mnt/data` directory would be accessible to the clients.
The above statement is also valid for all gateway backends.
## Explore Further
- [Minio Erasure Code QuickStart Guide](https://docs.minio.io/docs/minio-erasure-code-quickstart-guide)
- [Use `mc` with Minio Server](https://docs.minio.io/docs/minio-client-quickstart-guide)
@@ -109,3 +115,7 @@ Minio Server comes with an embedded web based object browser. Point your web bro
## Contribute to Minio Project
Please follow Minio [Contributor's Guide](https://github.com/minio/minio/blob/master/CONTRIBUTING.md)
## License
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fminio%2Fminio.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fminio%2Fminio?ref=badge_large)
-197
View File
@@ -1,197 +0,0 @@
# Minio 快速入门 [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![codecov](https://codecov.io/gh/minio/minio/branch/master/graph/badge.svg)](https://codecov.io/gh/minio/minio) [![Snap Status](https://build.snapcraft.io/badge/minio/minio.svg)](https://build.snapcraft.io/user/minio/minio)
Minio是一个对象存储服务,基于Apache License v2.0协议. 它完全兼容亚马逊的S3云储存服务,非常适合于存储很多非结构化的数据,例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等,而一个对象文件可以是任意大小,从几kb到最大5T不等。
## 1. 下载
Minio是一个非常轻量的服务,可以很简单的和其他应用的结合,类似 NodeJS, Redis 或者 MySQL.
| Platform| Architecture | URL|
| ----------| -------- | ------|
|GNU/Linux|64-bit Intel|https://dl.minio.io/server/minio/release/linux-amd64/minio|
|Apple OS X|64-bit Intel|https://dl.minio.io/server/minio/release/darwin-amd64/minio|
|Microsoft Windows|64-bit|https://dl.minio.io/server/minio/release/windows-amd64/minio.exe|
|FreeBSD|64-bit|https://dl.minio.io/server/minio/release/freebsd-amd64/minio|
### Homebrew 安装
使用[Homebrew](http://brew.sh/) 来安装minio
```sh
$ brew install minio
$ minio --help
```
### 源码安装
源码安装只针对开发者和一些高级用户,如果你还没有golang的环境,请安装golang官网安装[How to install Golang](https://docs.minio.io/docs/zh-CN/how-to-install-golang).
```sh
$ go get -u github.com/minio/minio
```
## 2. 运行Minio服务
### GNU/Linux
```sh
$ chmod +x minio
$ ./minio --help
$ ./minio server /data
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
### OS X
```sh
$ chmod 755 minio
$ ./minio --help
$ ./minio server /data
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
### Microsoft Windows
```sh
C:\Users\Username\Downloads> minio.exe --help
C:\Users\Username\Downloads> minio.exe server D:\Photos
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc.exe config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
### Docker
```sh
$ docker pull minio/minio
$ docker run -p 9000:9000 minio/minio
```
访问minio的docker入门指南获得更多内容 [here](https://docs.minio.io/docs/zh-CN/minio-docker-quickstart-guide)
### FreeBSD
```sh
$ chmod 755 minio
$ ./minio --help
$ ./minio server /data
端点: http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
区域: us-east-1
浏览器访问入口:
http://10.0.0.10:9000 http://127.0.0.1:9000 http://172.17.0.1:9000
命令行访问: https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide
$ mc config host add myminio http://10.0.0.10:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
对象操作API (兼容Amazon S3):
Go: https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide
Java: https://docs.minio.io/docs/zh-CN/java-client-quickstart-guide
Python: https://docs.minio.io/docs/zh-CN/python-client-quickstart-guide
JavaScript: https://docs.minio.io/docs/zh-CN/javascript-client-quickstart-guide
```
请访问FreeBSD的官网指南获取更多详细信息[here](https://www.freebsd.org/doc/handbook/zfs-quickstart.html)
## 3. 使用浏览器测试minio服务
打开浏览器并输入 http://127.0.0.1:9000 查看在minio服务器上面的所有bucket
![Screenshot](https://github.com/minio/minio/blob/master/docs/screenshots/minio-browser.jpg?raw=true)
## 4. 使用`mc`测试minio服务
按照 [这个](https://docs.minio.io/docs/minio-client-quickstart-guide) 安装mc. 使用 `mc ls` 命令显示所有在minio服务上面的bucket.
```sh
$ mc ls myminio/
[2015-08-05 08:13:22 IST] 0B andoria/
[2015-08-05 06:14:26 IST] 0B deflector/
[2015-08-05 08:13:11 IST] 0B ferenginar/
[2016-03-08 14:56:35 IST] 0B jarjarbing/
[2016-01-20 16:07:41 IST] 0B my.minio.io/
```
查看更多的例子请访问 [Minio Client Complete Guide](https://docs.minio.io/docs/zh-CN/minio-client-complete-guide).
## 5. 更多内容
- [Minio Erasure Code 快速入门](https://docs.minio.io/docs/zh-CN/minio-erasure-code-quickstart-guide)
- [Minio Docker 快速入门](https://docs.minio.io/docs/zh-CN/minio-docker-quickstart-guide)
- [使用`mc`测试 Minio Server](https://docs.minio.io/docs/zh-CN/minio-client-quickstart-guide)
- [使用 `aws-cli` 测试 Minio Server](https://docs.minio.io/docs/zh-CN/aws-cli-with-minio)
- [使用 `s3cmd` 测试 Minio Server](https://docs.minio.io/docs/zh-CN/s3cmd-with-minio)
- [使用 `minio-go` SDK ce's测试 Minio Server](https://docs.minio.io/docs/zh-CN/golang-client-quickstart-guide)
## 6. 给Minio项目贡献
请按照Minio [贡献者指导手册](https://github.com/minio/minio/blob/master/CONTRIBUTING.md)
+5
View File
@@ -104,6 +104,11 @@ go get -u github.com/minio/minio
## 使用Minio客户端 `mc`进行验证
`mc` 提供了一些UNIX常用命令的替代品,像ls, cat, cp, mirror, diff这些。 它支持文件系统和亚马逊S3云存储服务。 更多信息请参考 [mc快速入门](https://docs.minio.io/docs/minio-client-quickstart-guide) 。
## 已经存在的数据
当在单块磁盘上部署Minio server,Minio server允许客户端访问数据目录下已经存在的数据。比如,如果Minio使用`minio server /mnt/data`启动,那么所有已经在`/mnt/data`目录下的数据都可以被客户端访问到。
上述描述对所有网关后端同样有效。
## 了解更多
- [Minio纠删码入门](https://docs.minio.io/docs/minio-erasure-code-quickstart-guide)
- [`mc`快速入门](https://docs.minio.io/docs/minio-client-quickstart-guide)
+2
View File
@@ -41,6 +41,8 @@ import _Login from './js/components/Login.js'
import _Browse from './js/components/Browse.js'
import fontAwesome from 'font-awesome/css/font-awesome.css'
import MaterialDesignIconicFonts from 'material-design-iconic-font/dist/css/material-design-iconic-font.min.css'
import Web from './js/web'
window.Web = Web
+49 -1
View File
@@ -24,6 +24,8 @@ export const SET_CURRENT_BUCKET = 'SET_CURRENT_BUCKET'
export const SET_CURRENT_PATH = 'SET_CURRENT_PATH'
export const SET_BUCKETS = 'SET_BUCKETS'
export const ADD_BUCKET = 'ADD_BUCKET'
export const REMOVE_BUCKET = 'REMOVE_BUCKET'
export const SHOW_BUCKET_DROPDOWN = 'SHOW_BUCKET_DROPDOWN'
export const SET_VISIBLE_BUCKETS = 'SET_VISIBLE_BUCKETS'
export const SET_OBJECTS = 'SET_OBJECTS'
export const APPEND_OBJECTS = 'APPEND_OBJECTS'
@@ -173,6 +175,27 @@ export const addBucket = bucket => {
}
}
export const removeBucket = bucket => {
return {
type: REMOVE_BUCKET,
bucket
}
}
export const showBucketDropdown = bucket => {
return {
type: SHOW_BUCKET_DROPDOWN,
showBucketDropdown: true
}
}
export const hideBucketDropdown = bucket => {
return {
type: SHOW_BUCKET_DROPDOWN,
showBucketDropdown: false
}
}
export const showMakeBucketModal = () => {
return {
type: SHOW_MAKEBUCKET_MODAL,
@@ -314,6 +337,31 @@ export const selectBucket = (newCurrentBucket, prefix) => {
}
}
export const deleteBucket = (bucket) => {
return (dispatch, getState) => {
// DeleteBucket() RPC call will ONLY delete a bucket if it is empty of
// objects. This means a call can just be sent, as it is entirely reversable
// and won't do any permanent damage.
web.DeleteBucket({
bucketName: bucket
})
.then(() => {
dispatch(showAlert({
type: 'info',
message: `Bucket '${bucket}' has been deleted.`
}))
dispatch(removeBucket(bucket))
})
.catch(err => {
let message = err.message
dispatch(showAlert({
type: 'danger',
message: message
}))
})
}
}
export const listObjects = () => {
return (dispatch, getState) => {
const {buckets, currentBucket, currentPath, marker, objects, istruncated, web} = getState()
@@ -352,7 +400,6 @@ export const listObjects = () => {
export const selectPrefix = prefix => {
return (dispatch, getState) => {
const {currentBucket, web} = getState()
dispatch(resetObjects())
dispatch(setLoadPath(prefix))
web.ListObjects({
bucketName: currentBucket,
@@ -367,6 +414,7 @@ export const selectPrefix = prefix => {
object.name = object.name.replace(`${prefix}`, '');
return object
})
dispatch(resetObjects())
dispatch(appendObjects(
objects,
res.nextmarker,
+29 -3
View File
@@ -210,9 +210,19 @@ export default class Browse extends React.Component {
dispatch(actions.hideAbout())
}
toggleBucketDropdown(e) {
const {dispatch, showBucketDropdown} = this.props
if (showBucketDropdown) {
dispatch(actions.hideBucketDropdown())
} else {
dispatch(actions.showBucketDropdown())
}
}
showBucketPolicy(e) {
e.preventDefault()
const {dispatch} = this.props
this.toggleBucketDropdown(e)
dispatch(actions.showBucketPolicy())
}
@@ -222,10 +232,17 @@ export default class Browse extends React.Component {
dispatch(actions.hideBucketPolicy())
}
deleteBucket(e, bucket) {
e.preventDefault()
const {dispatch} = this.props
this.toggleBucketDropdown(e)
dispatch(actions.deleteBucket(bucket))
browserHistory.push(`${minioBrowserPrefix}/`)
}
uploadFile(e) {
e.preventDefault()
const {dispatch, buckets} = this.props
const {dispatch, buckets, currentBucket} = this.props
if (buckets.length === 0) {
dispatch(actions.showAlert({
type: 'danger',
@@ -233,6 +250,13 @@ export default class Browse extends React.Component {
}))
return
}
if (currentBucket === '') {
dispatch(actions.showAlert({
type: 'danger',
message: "Please choose a bucket before trying to upload files."
}))
return
}
let file = e.target.files[0]
e.target.value = null
this.xhr = new XMLHttpRequest()
@@ -577,7 +601,9 @@ export default class Browse extends React.Component {
<SideBar searchBuckets={ this.searchBuckets.bind(this) }
selectBucket={ this.selectBucket.bind(this) }
clickOutside={ this.hideSidebar.bind(this) }
showPolicy={ this.showBucketPolicy.bind(this) } />
showPolicy={ this.showBucketPolicy.bind(this) }
deleteBucket={ this.deleteBucket.bind(this) }
toggleBucketDropdown={ this.toggleBucketDropdown.bind(this) } />
<div className="fe-body">
<div className={ 'list-actions' + (classNames({
' list-actions-toggled': checkedObjects.length > 0
+16 -2
View File
@@ -21,8 +21,9 @@ import Scrollbars from 'react-custom-scrollbars/lib/Scrollbars'
import connect from 'react-redux/lib/components/connect'
import logo from '../../img/logo.svg'
import Dropdown from 'react-bootstrap/lib/Dropdown'
let SideBar = ({visibleBuckets, loadBucket, currentBucket, selectBucket, searchBuckets, sidebarStatus, clickOutside, showPolicy}) => {
let SideBar = ({visibleBuckets, loadBucket, currentBucket, selectBucket, searchBuckets, sidebarStatus, clickOutside, showPolicy, deleteBucket, toggleBucketDropdown, showBucketDropdown}) => {
const list = visibleBuckets.map((bucket, i) => {
return <li className={ classNames({
@@ -33,7 +34,19 @@ let SideBar = ({visibleBuckets, loadBucket, currentBucket, selectBucket, searchB
}) }>
{ bucket }
</a>
<i className="fesli-trigger" onClick={ showPolicy }></i>
<Dropdown open={bucket === currentBucket && showBucketDropdown} onToggle={toggleBucketDropdown} className="bucket-dropdown" id="bucket-dropdown">
<Dropdown.Toggle noCaret>
<i className="zmdi zmdi-more-vert" />
</Dropdown.Toggle>
<Dropdown.Menu className="dropdown-menu-right">
<li>
<a onClick={ showPolicy }>Edit policy</a>
</li>
<li>
<a onClick={ (e) => deleteBucket(e, bucket) }>Delete</a>
</li>
</Dropdown.Menu>
</Dropdown>
</li>
})
@@ -80,6 +93,7 @@ export default connect(state => {
visibleBuckets: state.visibleBuckets,
loadBucket: state.loadBucket,
currentBucket: state.currentBucket,
showBucketDropdown: state.showBucketDropdown,
sidebarStatus: state.sidebarStatus
}
})(SideBar)
+9
View File
@@ -25,6 +25,7 @@ export default (state = {
storageInfo: {},
serverInfo: {},
currentBucket: '',
showBucketDropdown: false,
currentPath: '',
showMakeBucketModal: false,
uploads: {},
@@ -71,6 +72,14 @@ export default (state = {
newState.buckets = [action.bucket, ...newState.buckets]
newState.visibleBuckets = [action.bucket, ...newState.visibleBuckets]
break
case actions.REMOVE_BUCKET:
newState.buckets = newState.buckets.filter(bucket => bucket != action.bucket)
newState.visibleBuckets = newState.visibleBuckets.filter(bucket => bucket != action.bucket)
newState.currentBucket = ""
break
case actions.SHOW_BUCKET_DROPDOWN:
newState.showBucketDropdown = action.showBucketDropdown
break
case actions.SET_VISIBLE_BUCKETS:
newState.visibleBuckets = action.visibleBuckets
break
+3
View File
@@ -87,6 +87,9 @@ export default class Web {
MakeBucket(args) {
return this.makeCall('MakeBucket', args)
}
DeleteBucket(args) {
return this.makeCall('DeleteBucket', args)
}
ListObjects(args) {
return this.makeCall('ListObjects', args)
}
+1 -1
View File
@@ -108,7 +108,7 @@ div.fesl-row {
&[data-type=zip] { .list-type(#427089, '\f1c6'); }
&[data-type=audio] { .list-type(#009688, '\f1c7'); }
&[data-type=code] { .list-type(#997867, "\f1c9"); }
&[data-type=excel] { .list-type(#f1c3, '\f1c3'); }
&[data-type=excel] { .list-type(#64c866, '\f1c3'); }
&[data-type=image] { .list-type(#f06292, '\f1c5'); }
&[data-type=video] { .list-type(#f8c363, '\f1c8'); }
&[data-type=other] { .list-type(#afafaf, '\f016'); }
+31 -8
View File
@@ -111,7 +111,7 @@
}
&:hover {
.fesli-trigger {
.bucket-dropdown .dropdown-toggle {
.opacity(0.6);
&:hover {
@@ -132,17 +132,40 @@
}
}
.fesli-trigger {
.opacity(0);
.transition(all);
.transition-duration(200ms);
/* Dropdown */
.bucket-dropdown {
position: absolute;
top: 0;
right: 0;
top: 0px;
right: 0px;
width: 35px;
height: 100%;
cursor: pointer;
background: url(../../img/more-h-light.svg) no-repeat top 20px left;
color: @white;
.dropdown-toggle {
.opacity(0);
.transition(all);
.transition-duration(200ms);
font-size: 20px;
background-color: transparent;
}
.dropdown-menu-right {
padding: 15px 0;
margin-top: -1px;
& li {
a:before {
content: none;
-webkit-box-shadow: @dropdown-shadow;
box-shadow: @dropdown-shadow;
}
&:not(.active):hover {
& > a {
color: @dropdown-link-hover-color;
}
}
}
}
}
/* Scrollbar */
+1
View File
@@ -60,6 +60,7 @@
"humanize": "0.0.9",
"json-loader": "^0.5.4",
"local-storage-fallback": "^1.3.0",
"material-design-iconic-font": "^2.2.0",
"mime-db": "^1.25.0",
"mime-types": "^2.1.13",
"moment": "^2.15.1",
+53 -49
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -21,7 +21,7 @@ _init() {
## Minimum required versions for build dependencies
GIT_VERSION="1.0"
GO_VERSION="1.9.1"
GO_VERSION="1.9.4"
OSX_VERSION="10.8"
KNAME=$(uname -s)
ARCH=$(uname -m)
+365 -584
View File
File diff suppressed because it is too large Load Diff
+349 -787
View File
File diff suppressed because it is too large Load Diff
+690
View File
@@ -0,0 +1,690 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"encoding/json"
"fmt"
"strings"
"sync"
"time"
"github.com/minio/minio/pkg/madmin"
)
// healStatusSummary - overall short summary of a healing sequence
type healStatusSummary string
// healStatusSummary constants
const (
healNotStartedStatus healStatusSummary = "not started"
healRunningStatus = "running"
healStoppedStatus = "stopped"
healFinishedStatus = "finished"
)
const (
// a heal sequence with this many un-consumed heal result
// items blocks until heal-status consumption resumes or is
// aborted due to timeout.
maxUnconsumedHealResultItems = 1000
// if no heal-results are consumed (via the heal-status API)
// for this timeout duration, the heal sequence is aborted.
healUnconsumedTimeout = 24 * time.Hour
// time-duration to keep heal sequence state after it
// completes.
keepHealSeqStateDuration = time.Minute * 10
)
var (
errHealIdleTimeout = fmt.Errorf("healing results were not consumed for too long")
errHealPushStopNDiscard = fmt.Errorf("heal push stopped due to heal stop signal")
errHealStopSignalled = fmt.Errorf("heal stop signalled")
errFnHealFromAPIErr = func(err error) error {
errCode := toAPIErrorCode(err)
apiErr := getAPIError(errCode)
return fmt.Errorf("Heal internal error: %s: %s",
apiErr.Code, apiErr.Description)
}
)
// healSequenceStatus - accumulated status of the heal sequence
type healSequenceStatus struct {
// lock to update this structure as it is concurrently
// accessed
updateLock *sync.RWMutex
// summary and detail for failures
Summary healStatusSummary `json:"Summary"`
FailureDetail string `json:"Detail,omitempty"`
StartTime time.Time `json:"StartTime"`
// disk information
NumDisks int `json:"NumDisks"`
// settings for the heal sequence
HealSettings madmin.HealOpts `json:"Settings"`
// slice of available heal result records
Items []madmin.HealResultItem `json:"Items"`
}
// structure to hold state of all heal sequences in server memory
type allHealState struct {
sync.Mutex
// map of heal path to heal sequence
healSeqMap map[string]*healSequence
}
var (
// global server heal state
globalAllHealState allHealState
)
// initAllHealState - initialize healing apparatus
func initAllHealState(isErasureMode bool) {
if !isErasureMode {
return
}
globalAllHealState = allHealState{
healSeqMap: make(map[string]*healSequence),
}
}
// getHealSequence - Retrieve a heal sequence by path. The second
// argument returns if a heal sequence actually exists.
func (ahs *allHealState) getHealSequence(path string) (h *healSequence, exists bool) {
ahs.Lock()
defer ahs.Unlock()
h, exists = ahs.healSeqMap[path]
return h, exists
}
// LaunchNewHealSequence - launches a background routine that performs
// healing according to the healSequence argument. For each heal
// sequence, state is stored in the `globalAllHealState`, which is a
// map of the heal path to `healSequence` which holds state about the
// heal sequence.
//
// Heal results are persisted in server memory for
// `keepHealSeqStateDuration`. This function also launches a
// background routine to clean up heal results after the
// aforementioned duration.
func (ahs *allHealState) LaunchNewHealSequence(h *healSequence) (
respBytes []byte, errCode APIErrorCode, errMsg string) {
existsAndLive := false
he, exists := ahs.getHealSequence(h.path)
if exists {
if !he.hasEnded() || len(he.currentStatus.Items) > 0 {
existsAndLive = true
}
}
if existsAndLive {
// A heal sequence exists on the given path.
if h.forceStarted {
// stop the running heal sequence - wait for
// it to finish.
he.stop()
for !he.hasEnded() {
time.Sleep(10 * time.Second)
}
} else {
errMsg = "Heal is already running on the given path " +
"(use force-start option to stop and start afresh). " +
fmt.Sprintf("The heal was started by IP %s at %s",
h.clientAddress, h.startTime)
return nil, ErrHealAlreadyRunning, errMsg
}
}
ahs.Lock()
defer ahs.Unlock()
// Check if new heal sequence to be started overlaps with any
// existing, running sequence
for k, hSeq := range ahs.healSeqMap {
if !hSeq.hasEnded() && (strings.HasPrefix(k, h.path) ||
strings.HasPrefix(h.path, k)) {
errMsg = "The provided heal sequence path overlaps with an existing " +
fmt.Sprintf("heal path: %s", k)
return nil, ErrHealOverlappingPaths, errMsg
}
}
// Add heal state and start sequence
ahs.healSeqMap[h.path] = h
// Launch top-level background heal go-routine
go h.healSequenceStart()
// Launch clean-up routine to remove this heal sequence (after
// it ends) from the global state after timeout has elapsed.
go func() {
var keepStateTimeout <-chan time.Time
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
everyMinute := ticker.C
for {
select {
// Check every minute if heal sequence has ended.
case <-everyMinute:
if h.hasEnded() {
keepStateTimeout = time.After(keepHealSeqStateDuration)
everyMinute = nil
}
// This case does not fire until the heal
// sequence completes.
case <-keepStateTimeout:
// Heal sequence has ended, keep
// results state duration has elapsed,
// so purge state.
ahs.Lock()
defer ahs.Unlock()
delete(ahs.healSeqMap, h.path)
return
case <-globalServiceDoneCh:
// server could be restarting - need
// to exit immediately
return
}
}
}()
b, err := json.Marshal(madmin.HealStartSuccess{
h.clientToken,
h.clientAddress,
h.startTime,
})
if err != nil {
errorIf(err, "Failed to marshal heal result into json.")
return nil, ErrInternalError, ""
}
return b, ErrNone, ""
}
// PopHealStatusJSON - Called by heal-status API. It fetches the heal
// status results from global state and returns its JSON
// representation. The clientToken helps ensure there aren't
// conflicting clients fetching status.
func (ahs *allHealState) PopHealStatusJSON(path string,
clientToken string) ([]byte, APIErrorCode) {
// fetch heal state for given path
h, exists := ahs.getHealSequence(path)
if !exists {
// If there is no such heal sequence, return error.
return nil, ErrHealNoSuchProcess
}
// Check if client-token is valid
if clientToken != h.clientToken {
return nil, ErrHealInvalidClientToken
}
// Take lock to access and update the heal-sequence
h.currentStatus.updateLock.Lock()
defer h.currentStatus.updateLock.Unlock()
numItems := len(h.currentStatus.Items)
// calculate index of most recently available heal result
// record.
lastResultIndex := h.lastSentResultIndex
if numItems > 0 {
lastResultIndex = h.currentStatus.Items[numItems-1].ResultIndex
}
// After sending status to client, and before relinquishing
// the updateLock, reset Item to nil and record the result
// index sent to the client.
defer func(i int64) {
h.lastSentResultIndex = i
h.currentStatus.Items = nil
}(lastResultIndex)
jbytes, err := json.Marshal(h.currentStatus)
if err != nil {
errorIf(err, "Failed to marshal heal result into json.")
return nil, ErrInternalError
}
return jbytes, ErrNone
}
// healSequence - state for each heal sequence initiated on the
// server.
type healSequence struct {
// bucket, and prefix on which heal seq. was initiated
bucket, objPrefix string
// path is just bucket + "/" + objPrefix
path string
// time at which heal sequence was started
startTime time.Time
// Heal client info
clientToken, clientAddress string
// was this heal sequence force started?
forceStarted bool
// heal settings applied to this heal sequence
settings madmin.HealOpts
// current accumulated status of the heal sequence
currentStatus healSequenceStatus
// channel signalled by background routine when traversal has
// completed
traverseAndHealDoneCh chan error
// channel to signal heal sequence to stop (e.g. from the
// heal-stop API)
stopSignalCh chan struct{}
// the last result index sent to client
lastSentResultIndex int64
}
// NewHealSequence - creates healSettings, assumes bucket and
// objPrefix are already validated.
func newHealSequence(bucket, objPrefix, clientAddr string,
numDisks int, hs madmin.HealOpts, forceStart bool) *healSequence {
return &healSequence{
bucket: bucket,
objPrefix: objPrefix,
path: bucket + "/" + objPrefix,
startTime: UTCNow(),
clientToken: mustGetUUID(),
clientAddress: clientAddr,
forceStarted: forceStart,
settings: hs,
currentStatus: healSequenceStatus{
Summary: healNotStartedStatus,
HealSettings: hs,
NumDisks: numDisks,
updateLock: &sync.RWMutex{},
},
traverseAndHealDoneCh: make(chan error),
stopSignalCh: make(chan struct{}),
}
}
// isQuitting - determines if the heal sequence is quitting (due to an
// external signal)
func (h *healSequence) isQuitting() bool {
select {
case <-h.stopSignalCh:
return true
default:
return false
}
}
// check if the heal sequence has ended
func (h *healSequence) hasEnded() bool {
h.currentStatus.updateLock.RLock()
summary := h.currentStatus.Summary
h.currentStatus.updateLock.RUnlock()
return summary == healStoppedStatus || summary == healFinishedStatus
}
// stops the heal sequence - safe to call multiple times.
func (h *healSequence) stop() {
select {
case <-h.stopSignalCh:
default:
close(h.stopSignalCh)
}
}
// pushHealResultItem - pushes a heal result item for consumption in
// the heal-status API. It blocks if there are
// maxUnconsumedHealResultItems. When it blocks, the heal sequence
// routine is effectively paused - this happens when the server has
// accumulated the maximum number of heal records per heal
// sequence. When the client consumes further records, the heal
// sequence automatically resumes. The return value indicates if the
// operation succeeded.
func (h *healSequence) pushHealResultItem(r madmin.HealResultItem) error {
// start a timer to keep an upper time limit to find an empty
// slot to add the given heal result - if no slot is found it
// means that the server is holding the maximum amount of
// heal-results in memory and the client has not consumed it
// for too long.
unconsumedTimer := time.NewTimer(healUnconsumedTimeout)
defer func() {
// stop the timeout timer so it is garbage collected.
if !unconsumedTimer.Stop() {
<-unconsumedTimer.C
}
}()
var itemsLen int
for {
h.currentStatus.updateLock.Lock()
itemsLen = len(h.currentStatus.Items)
if itemsLen == maxUnconsumedHealResultItems {
// unlock and wait to check again if we can push
h.currentStatus.updateLock.Unlock()
// wait for a second, or quit if an external
// stop signal is received or the
// unconsumedTimer fires.
select {
// Check after a second
case <-time.After(time.Second):
continue
case <-h.stopSignalCh:
// discard result and return.
return errHealPushStopNDiscard
// Timeout if no results consumed for too
// long.
case <-unconsumedTimer.C:
return errHealIdleTimeout
}
}
break
}
// Set the correct result index for the new result item
if itemsLen > 0 {
r.ResultIndex = 1 + h.currentStatus.Items[itemsLen-1].ResultIndex
} else {
r.ResultIndex = 1 + h.lastSentResultIndex
}
// append to results
h.currentStatus.Items = append(h.currentStatus.Items, r)
// release lock
h.currentStatus.updateLock.Unlock()
// This is a "safe" point for the heal sequence to quit if
// signalled externally.
if h.isQuitting() {
return errHealStopSignalled
}
return nil
}
// healSequenceStart - this is the top-level background heal
// routine. It launches another go-routine that actually traverses
// on-disk data, checks and heals according to the selected
// settings. This go-routine itself, (1) monitors the traversal
// routine for completion, and (2) listens for external stop
// signals. When either event happens, it sets the finish status for
// the heal-sequence.
func (h *healSequence) healSequenceStart() {
// Set status as running
h.currentStatus.updateLock.Lock()
h.currentStatus.Summary = healRunningStatus
h.currentStatus.StartTime = UTCNow()
h.currentStatus.updateLock.Unlock()
go h.traverseAndHeal()
select {
case err, ok := <-h.traverseAndHealDoneCh:
h.currentStatus.updateLock.Lock()
defer h.currentStatus.updateLock.Unlock()
// Heal traversal is complete.
if ok {
// heal traversal had an error.
h.currentStatus.Summary = healStoppedStatus
h.currentStatus.FailureDetail = err.Error()
} else {
// heal traversal succeeded.
h.currentStatus.Summary = healFinishedStatus
}
case <-h.stopSignalCh:
h.currentStatus.updateLock.Lock()
h.currentStatus.Summary = healStoppedStatus
h.currentStatus.FailureDetail = errHealStopSignalled.Error()
h.currentStatus.updateLock.Unlock()
// drain traverse channel so the traversal
// go-routine does not leak.
go func() {
// Eventually the traversal go-routine closes
// the channel and returns, so this go-routine
// itself will not leak.
<-h.traverseAndHealDoneCh
}()
}
}
// traverseAndHeal - traverses on-disk data and performs healing
// according to settings. At each "safe" point it also checks if an
// external quit signal has been received and quits if so. Since the
// healing traversal may be mutating on-disk data when an external
// quit signal is received, this routine cannot quit immediately and
// has to wait until a safe point is reached, such as between scanning
// two objects.
func (h *healSequence) traverseAndHeal() {
var err error
checkErr := func(f func() error) {
switch {
case err != nil:
return
case h.isQuitting():
err = errHealStopSignalled
return
}
err = f()
}
// Start with format healing
checkErr(h.healDiskFormat)
// Heal buckets and objects
checkErr(h.healBuckets)
if err != nil {
h.traverseAndHealDoneCh <- err
}
close(h.traverseAndHealDoneCh)
}
// healDiskFormat - heals format.json, return value indicates if a
// failure error occurred.
func (h *healSequence) healDiskFormat() error {
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return errServerNotInitialized
}
// Acquire lock on format.json
formatLock := globalNSMutex.NewNSLock(minioMetaBucket, formatConfigFile)
if err := formatLock.GetLock(globalHealingTimeout); err != nil {
return errFnHealFromAPIErr(err)
}
defer formatLock.Unlock()
// Create a new set of storage instances to heal format.json.
bootstrapDisks, err := initStorageDisks(globalEndpoints)
if err != nil {
return errFnHealFromAPIErr(err)
}
// Wrap into retrying disks
retryingDisks := initRetryableStorageDisks(bootstrapDisks,
time.Millisecond, time.Millisecond*5,
globalStorageHealthCheckInterval, globalStorageRetryThreshold)
// Heal format.json on available storage.
hres, err := healFormatXL(retryingDisks, h.settings.DryRun)
if err != nil {
return errFnHealFromAPIErr(err)
}
// reload object layer global only if we healed some disk
onlineBefore, onlineAfter := hres.GetOnlineCounts()
numHealed := onlineAfter - onlineBefore
if numHealed > 0 {
// Instantiate new object layer with newly formatted
// storage.
newObjectAPI, err := newXLObjects(retryingDisks)
if err != nil {
return errFnHealFromAPIErr(err)
}
// Set object layer with newly formatted storage to
// globalObjectAPI.
globalObjLayerMutex.Lock()
globalObjectAPI = newObjectAPI
globalObjLayerMutex.Unlock()
// Shutdown storage belonging to old object layer
// instance.
objectAPI.Shutdown()
// Inform peers to reinitialize storage with newly
// formatted storage.
reInitPeerDisks(globalAdminPeers)
}
// Push format heal result
return h.pushHealResultItem(hres)
}
// healBuckets - check for all buckets heal or just particular bucket.
func (h *healSequence) healBuckets() error {
// 1. If a bucket was specified, heal only the bucket.
if h.bucket != "" {
return h.healBucket(h.bucket)
}
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return errServerNotInitialized
}
buckets, err := objectAPI.ListBucketsHeal()
if err != nil {
return errFnHealFromAPIErr(err)
}
for _, bucket := range buckets {
err = h.healBucket(bucket.Name)
if err != nil {
return err
}
}
return nil
}
// healBucket - traverses and heals given bucket
func (h *healSequence) healBucket(bucket string) error {
if h.isQuitting() {
return errHealStopSignalled
}
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return errServerNotInitialized
}
results, err := objectAPI.HealBucket(bucket, h.settings.DryRun)
// push any available results before checking for error
for _, result := range results {
if perr := h.pushHealResultItem(result); perr != nil {
return perr
}
}
// handle heal-bucket error
if err != nil {
return err
}
if !h.settings.Recursive {
if h.objPrefix != "" {
// Check if an object named as the objPrefix exists,
// and if so heal it.
_, err = objectAPI.GetObjectInfo(bucket, h.objPrefix)
if err == nil {
err = h.healObject(bucket, h.objPrefix)
if err != nil {
return err
}
}
}
return nil
}
marker := ""
isTruncated := true
for isTruncated {
objectInfos, err := objectAPI.ListObjectsHeal(bucket,
h.objPrefix, marker, "", 1000)
if err != nil {
return errFnHealFromAPIErr(err)
}
for _, o := range objectInfos.Objects {
if err := h.healObject(o.Bucket, o.Name); err != nil {
return err
}
}
isTruncated = objectInfos.IsTruncated
marker = objectInfos.NextMarker
}
return nil
}
// healObject - heal the given object and record result
func (h *healSequence) healObject(bucket, object string) error {
if h.isQuitting() {
return errHealStopSignalled
}
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return errServerNotInitialized
}
hri, err := objectAPI.HealObject(bucket, object, h.settings.DryRun)
if err != nil {
hri.Detail = err.Error()
}
return h.pushHealResultItem(hri)
}
+30 -28
View File
@@ -16,7 +16,15 @@
package cmd
import router "github.com/gorilla/mux"
import (
"net/http"
router "github.com/gorilla/mux"
)
const (
adminAPIPathPrefix = "/minio/admin"
)
// adminAPIHandlers provides HTTP handlers for Minio admin API.
type adminAPIHandlers struct {
@@ -27,50 +35,44 @@ func registerAdminRouter(mux *router.Router) {
adminAPI := adminAPIHandlers{}
// Admin router
adminRouter := mux.NewRoute().PathPrefix("/").Subrouter()
adminRouter := mux.NewRoute().PathPrefix(adminAPIPathPrefix).Subrouter()
// Version handler
adminRouter.Methods(http.MethodGet).Path("/version").HandlerFunc(adminAPI.VersionHandler)
adminV1Router := adminRouter.PathPrefix("/v1").Subrouter()
/// Service operations
// Service status
adminRouter.Methods("GET").Queries("service", "").Headers(minioAdminOpHeader, "status").HandlerFunc(adminAPI.ServiceStatusHandler)
adminV1Router.Methods(http.MethodGet).Path("/service").HandlerFunc(adminAPI.ServiceStatusHandler)
// Service restart
adminRouter.Methods("POST").Queries("service", "").Headers(minioAdminOpHeader, "restart").HandlerFunc(adminAPI.ServiceRestartHandler)
// Service update credentials
adminRouter.Methods("POST").Queries("service", "").Headers(minioAdminOpHeader, "set-credentials").HandlerFunc(adminAPI.ServiceCredentialsHandler)
// Service restart and stop - TODO
adminV1Router.Methods(http.MethodPost).Path("/service").HandlerFunc(adminAPI.ServiceStopNRestartHandler)
// Info operations
adminRouter.Methods("GET").Queries("info", "").HandlerFunc(adminAPI.ServerInfoHandler)
adminV1Router.Methods(http.MethodGet).Path("/info").HandlerFunc(adminAPI.ServerInfoHandler)
/// Lock operations
// List Locks
adminRouter.Methods("GET").Queries("lock", "").Headers(minioAdminOpHeader, "list").HandlerFunc(adminAPI.ListLocksHandler)
adminV1Router.Methods(http.MethodGet).Path("/locks").HandlerFunc(adminAPI.ListLocksHandler)
// Clear locks
adminRouter.Methods("POST").Queries("lock", "").Headers(minioAdminOpHeader, "clear").HandlerFunc(adminAPI.ClearLocksHandler)
adminV1Router.Methods(http.MethodDelete).Path("/locks").HandlerFunc(adminAPI.ClearLocksHandler)
/// Heal operations
// List Objects needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-objects").HandlerFunc(adminAPI.ListObjectsHealHandler)
// List Uploads needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-uploads").HandlerFunc(adminAPI.ListUploadsHealHandler)
// List Buckets needing heal.
adminRouter.Methods("GET").Queries("heal", "").Headers(minioAdminOpHeader, "list-buckets").HandlerFunc(adminAPI.ListBucketsHealHandler)
// Heal Buckets.
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "bucket").HandlerFunc(adminAPI.HealBucketHandler)
// Heal Objects.
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "object").HandlerFunc(adminAPI.HealObjectHandler)
// Heal Format.
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "format").HandlerFunc(adminAPI.HealFormatHandler)
// Heal Uploads.
adminRouter.Methods("POST").Queries("heal", "").Headers(minioAdminOpHeader, "upload").HandlerFunc(adminAPI.HealUploadHandler)
// Heal processing endpoint.
adminV1Router.Methods(http.MethodPost).Path("/heal/").HandlerFunc(adminAPI.HealHandler)
adminV1Router.Methods(http.MethodPost).Path("/heal/{bucket}").HandlerFunc(adminAPI.HealHandler)
adminV1Router.Methods(http.MethodPost).Path("/heal/{bucket}/{prefix:.*}").HandlerFunc(adminAPI.HealHandler)
/// Config operations
// Update credentials
adminV1Router.Methods(http.MethodPut).Path("/config/credential").HandlerFunc(adminAPI.UpdateCredentialsHandler)
// Get config
adminRouter.Methods("GET").Queries("config", "").Headers(minioAdminOpHeader, "get").HandlerFunc(adminAPI.GetConfigHandler)
// Set Config
adminRouter.Methods("PUT").Queries("config", "").Headers(minioAdminOpHeader, "set").HandlerFunc(adminAPI.SetConfigHandler)
adminV1Router.Methods(http.MethodGet).Path("/config").HandlerFunc(adminAPI.GetConfigHandler)
// Set config
adminV1Router.Methods(http.MethodPut).Path("/config").HandlerFunc(adminAPI.SetConfigHandler)
}
+35 -24
View File
@@ -23,7 +23,6 @@ import (
"os"
"path"
"path/filepath"
"reflect"
"sort"
"sync"
"time"
@@ -34,7 +33,7 @@ import (
const (
// Admin service names
serviceRestartRPC = "Admin.Restart"
signalServiceRPC = "Admin.SignalService"
listLocksRPC = "Admin.ListLocks"
reInitDisksRPC = "Admin.ReInitDisks"
serverInfoDataRPC = "Admin.ServerInfoData"
@@ -56,7 +55,7 @@ type remoteAdminClient struct {
// adminCmdRunner - abstracts local and remote execution of admin
// commands like service stop and service restart.
type adminCmdRunner interface {
Restart() error
SignalService(s serviceSignal) error
ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error)
ReInitDisks() error
ServerInfoData() (ServerInfoData, error)
@@ -65,10 +64,16 @@ type adminCmdRunner interface {
CommitConfig(tmpFileName string) error
}
// Restart - Sends a message over channel to the go-routine
// responsible for restarting the process.
func (lc localAdminClient) Restart() error {
globalServiceSignalCh <- serviceRestart
var errUnsupportedSignal = fmt.Errorf("unsupported signal: only restart and stop signals are supported")
// SignalService - sends a restart or stop signal to the local server
func (lc localAdminClient) SignalService(s serviceSignal) error {
switch s {
case serviceRestart, serviceStop:
globalServiceSignalCh <- s
default:
return errUnsupportedSignal
}
return nil
}
@@ -77,25 +82,31 @@ func (lc localAdminClient) ListLocks(bucket, prefix string, duration time.Durati
return listLocksInfo(bucket, prefix, duration), nil
}
// Restart - Sends restart command to remote server via RPC.
func (rc remoteAdminClient) Restart() error {
args := AuthRPCArgs{}
reply := AuthRPCReply{}
return rc.Call(serviceRestartRPC, &args, &reply)
func (rc remoteAdminClient) SignalService(s serviceSignal) (err error) {
switch s {
case serviceRestart, serviceStop:
reply := AuthRPCReply{}
err = rc.Call(signalServiceRPC, &SignalServiceArgs{Sig: s},
&reply)
default:
err = errUnsupportedSignal
}
return err
}
// ListLocks - Sends list locks command to remote server via RPC.
func (rc remoteAdminClient) ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
listArgs := ListLocksQuery{
bucket: bucket,
prefix: prefix,
duration: duration,
Bucket: bucket,
Prefix: prefix,
Duration: duration,
}
var reply ListLocksReply
if err := rc.Call(listLocksRPC, &listArgs, &reply); err != nil {
return nil, err
}
return reply.volLocks, nil
return reply.VolLocks, nil
}
// ReInitDisks - There is nothing to do here, heal format REST API
@@ -225,7 +236,7 @@ func (rc remoteAdminClient) CommitConfig(tmpFileName string) error {
return nil
}
// adminPeer - represents an entity that implements Restart methods.
// adminPeer - represents an entity that implements admin API RPCs.
type adminPeer struct {
addr string
cmdRunner adminCmdRunner
@@ -274,11 +285,11 @@ func initGlobalAdminPeers(endpoints EndpointList) {
globalAdminPeers = makeAdminPeers(endpoints)
}
// invokeServiceCmd - Invoke Restart command.
// invokeServiceCmd - Invoke Restart/Stop command.
func invokeServiceCmd(cp adminPeer, cmd serviceSignal) (err error) {
switch cmd {
case serviceRestart:
err = cp.cmdRunner.Restart()
case serviceRestart, serviceStop:
err = cp.cmdRunner.SignalService(cmd)
}
return err
}
@@ -466,7 +477,7 @@ func getPeerConfig(peers adminPeers) ([]byte, error) {
// Find the maximally occurring config among peers in a
// distributed setup.
serverConfigs := make([]serverConfigV13, len(peers))
serverConfigs := make([]serverConfig, len(peers))
for i, configBytes := range configs {
if errs[i] != nil {
continue
@@ -493,7 +504,7 @@ func getPeerConfig(peers adminPeers) ([]byte, error) {
// getValidServerConfig - finds the server config that is present in
// quorum or more number of servers.
func getValidServerConfig(serverConfigs []serverConfigV13, errs []error) (scv serverConfigV13, e error) {
func getValidServerConfig(serverConfigs []serverConfig, errs []error) (scv serverConfig, e error) {
// majority-based quorum
quorum := len(serverConfigs)/2 + 1
@@ -536,7 +547,7 @@ func getValidServerConfig(serverConfigs []serverConfigV13, errs []error) (scv se
// seen. See example above for
// clarity.
continue
} else if j < i && reflect.DeepEqual(serverConfigs[i], serverConfigs[j]) {
} else if j < i && serverConfigs[i].ConfigDiff(&serverConfigs[j]) == "" {
// serverConfigs[i] is equal to
// serverConfigs[j], update
// serverConfigs[j]'s counter since it
@@ -555,7 +566,7 @@ func getValidServerConfig(serverConfigs []serverConfigV13, errs []error) (scv se
// We find the maximally occurring server config and check if
// there is quorum.
var configJSON serverConfigV13
var configJSON serverConfig
maxOccurrence := 0
for i, count := range configCounter {
if maxOccurrence < count {
+4 -4
View File
@@ -220,7 +220,7 @@ var (
// TestGetValidServerConfig - test for getValidServerConfig.
func TestGetValidServerConfig(t *testing.T) {
var c1, c2 serverConfigV13
var c1, c2 serverConfig
err := json.Unmarshal(config1, &c1)
if err != nil {
t.Fatalf("json unmarshal of %s failed: %v", string(config1), err)
@@ -233,7 +233,7 @@ func TestGetValidServerConfig(t *testing.T) {
// Valid config.
noErrs := []error{nil, nil, nil, nil}
serverConfigs := []serverConfigV13{c1, c2, c1, c1}
serverConfigs := []serverConfig{c1, c2, c1, c1}
validConfig, err := getValidServerConfig(serverConfigs, noErrs)
if err != nil {
t.Errorf("Expected a valid config but received %v instead", err)
@@ -244,7 +244,7 @@ func TestGetValidServerConfig(t *testing.T) {
}
// Invalid config - no quorum.
serverConfigs = []serverConfigV13{c1, c2, c2, c1}
serverConfigs = []serverConfig{c1, c2, c2, c1}
_, err = getValidServerConfig(serverConfigs, noErrs)
if err != errXLWriteQuorum {
t.Errorf("Expected to fail due to lack of quorum but received %v", err)
@@ -252,7 +252,7 @@ func TestGetValidServerConfig(t *testing.T) {
// All errors
allErrs := []error{errDiskNotFound, errDiskNotFound, errDiskNotFound, errDiskNotFound}
serverConfigs = []serverConfigV13{{}, {}, {}, {}}
serverConfigs = []serverConfig{{}, {}, {}, {}}
_, err = getValidServerConfig(serverConfigs, allErrs)
if err != errXLWriteQuorum {
t.Errorf("Expected to fail due to lack of quorum but received %v", err)
+15 -9
View File
@@ -38,18 +38,24 @@ type adminCmd struct {
AuthRPCServer
}
// SignalServiceArgs - provides the signal argument to SignalService RPC
type SignalServiceArgs struct {
AuthRPCArgs
Sig serviceSignal
}
// ListLocksQuery - wraps ListLocks API's query values to send over RPC.
type ListLocksQuery struct {
AuthRPCArgs
bucket string
prefix string
duration time.Duration
Bucket string
Prefix string
Duration time.Duration
}
// ListLocksReply - wraps ListLocks response over RPC.
type ListLocksReply struct {
AuthRPCReply
volLocks []VolumeLockInfo
VolLocks []VolumeLockInfo
}
// ServerInfoDataReply - wraps the server info response over RPC.
@@ -64,13 +70,13 @@ type ConfigReply struct {
Config []byte // json-marshalled bytes of serverConfigV13
}
// Restart - Restart this instance of minio server.
func (s *adminCmd) Restart(args *AuthRPCArgs, reply *AuthRPCReply) error {
// SignalService - Send a restart or stop signal to the service
func (s *adminCmd) SignalService(args *SignalServiceArgs, reply *AuthRPCReply) error {
if err := args.IsAuthenticated(); err != nil {
return err
}
globalServiceSignalCh <- serviceRestart
globalServiceSignalCh <- args.Sig
return nil
}
@@ -79,8 +85,8 @@ func (s *adminCmd) ListLocks(query *ListLocksQuery, reply *ListLocksReply) error
if err := query.IsAuthenticated(); err != nil {
return err
}
volLocks := listLocksInfo(query.bucket, query.prefix, query.duration)
*reply = ListLocksReply{volLocks: volLocks}
volLocks := listLocksInfo(query.Bucket, query.Prefix, query.Duration)
*reply = ListLocksReply{VolLocks: volLocks}
return nil
}
+37 -11
View File
@@ -23,8 +23,8 @@ import (
)
func testAdminCmd(cmd cmdType, t *testing.T) {
// reset globals.
// this is to make sure that the tests are not affected by modified globals.
// reset globals. this is to make sure that the tests are not
// affected by modified globals.
resetTestGlobals()
rootPath, err := newTestConfig(globalMinioDefaultRegion)
@@ -42,7 +42,7 @@ func testAdminCmd(cmd cmdType, t *testing.T) {
adminServer := adminCmd{}
args := LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
RequestTime: UTCNow(),
}
err = adminServer.Login(&args, &LoginRPCReply{})
@@ -55,12 +55,23 @@ func testAdminCmd(cmd cmdType, t *testing.T) {
<-globalServiceSignalCh
}()
ga := AuthRPCArgs{AuthToken: token}
sa := SignalServiceArgs{
AuthRPCArgs: AuthRPCArgs{AuthToken: token, Version: globalRPCAPIVersion},
Sig: cmd.toServiceSignal(),
}
genReply := AuthRPCReply{}
switch cmd {
case restartCmd:
if err = adminServer.Restart(&ga, &genReply); err != nil {
t.Errorf("restartCmd: Expected: <nil>, got: %v", err)
case restartCmd, stopCmd:
if err = adminServer.SignalService(&sa, &genReply); err != nil {
t.Errorf("restartCmd/stopCmd: Expected: <nil>, got: %v",
err)
}
default:
err = adminServer.SignalService(&sa, &genReply)
if err != nil && err.Error() != errUnsupportedSignal.Error() {
t.Errorf("invalidSignal %s: unexpected error got: %v",
cmd, err)
}
}
}
@@ -70,6 +81,16 @@ func TestAdminRestart(t *testing.T) {
testAdminCmd(restartCmd, t)
}
// TestAdminStop - test for Admin.Stop RPC service.
func TestAdminStop(t *testing.T) {
testAdminCmd(stopCmd, t)
}
// TestAdminStatus - test for Admin.Status RPC service (error case)
func TestAdminStatus(t *testing.T) {
testAdminCmd(statusCmd, t)
}
// TestReInitDisks - test for Admin.ReInitDisks RPC service.
func TestReInitDisks(t *testing.T) {
// Reset global variables to start afresh.
@@ -103,7 +124,7 @@ func TestReInitDisks(t *testing.T) {
args := LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
RequestTime: UTCNow(),
}
err = adminServer.Login(&args, &LoginRPCReply{})
@@ -113,6 +134,7 @@ func TestReInitDisks(t *testing.T) {
authArgs := AuthRPCArgs{
AuthToken: token,
Version: globalRPCAPIVersion,
}
authReply := AuthRPCReply{}
@@ -130,7 +152,7 @@ func TestReInitDisks(t *testing.T) {
fsAdminServer := adminCmd{}
fsArgs := LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
RequestTime: UTCNow(),
}
fsReply := LoginRPCReply{}
@@ -141,6 +163,7 @@ func TestReInitDisks(t *testing.T) {
authArgs = AuthRPCArgs{
AuthToken: token,
Version: globalRPCAPIVersion,
}
authReply = AuthRPCReply{}
// Attempt ReInitDisks service on a FS backend.
@@ -172,7 +195,7 @@ func TestGetConfig(t *testing.T) {
args := LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
RequestTime: UTCNow(),
}
reply := LoginRPCReply{}
@@ -183,6 +206,7 @@ func TestGetConfig(t *testing.T) {
authArgs := AuthRPCArgs{
AuthToken: token,
Version: globalRPCAPIVersion,
}
configReply := ConfigReply{}
@@ -219,7 +243,7 @@ func TestWriteAndCommitConfig(t *testing.T) {
}
args := LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
RequestTime: UTCNow(),
}
reply := LoginRPCReply{}
@@ -234,6 +258,7 @@ func TestWriteAndCommitConfig(t *testing.T) {
wArgs := WriteConfigArgs{
AuthRPCArgs: AuthRPCArgs{
AuthToken: token,
Version: globalRPCAPIVersion,
},
TmpFileName: tmpFileName,
Buf: buf,
@@ -251,6 +276,7 @@ func TestWriteAndCommitConfig(t *testing.T) {
cArgs := CommitConfigArgs{
AuthRPCArgs: AuthRPCArgs{
AuthToken: token,
Version: globalRPCAPIVersion,
},
FileName: tmpFileName,
}
+71 -2
View File
@@ -18,6 +18,7 @@ package cmd
import (
"encoding/xml"
"fmt"
"net/http"
"github.com/minio/minio/pkg/auth"
@@ -40,8 +41,8 @@ type APIErrorResponse struct {
Key string
BucketName string
Resource string
RequestID string `xml:"RequestId"`
HostID string `xml:"HostId"`
RequestID string `xml:"RequestId" json:"RequestId"`
HostID string `xml:"HostId" json:"HostId"`
}
// APIErrorCode type of error status.
@@ -123,6 +124,7 @@ const (
ErrUnsupportedMetadata
ErrMaximumExpires
ErrSlowDown
ErrInvalidPrefixMarker
// Add new error codes here.
// Server-Side-Encryption (with Customer provided key) related API errors.
@@ -156,6 +158,7 @@ const (
ErrReadQuorum
ErrWriteQuorum
ErrStorageFull
ErrRequestBodyParse
ErrObjectExistsAsDirectory
ErrPolicyNesting
ErrInvalidObjectName
@@ -172,12 +175,21 @@ const (
// Please open a https://github.com/minio/minio/issues before adding
// new error codes here.
ErrMalformedJSON
ErrAdminInvalidAccessKey
ErrAdminInvalidSecretKey
ErrAdminConfigNoQuorum
ErrAdminConfigTooLarge
ErrAdminConfigBadJSON
ErrAdminCredentialsMismatch
ErrInsecureClientRequest
ErrObjectTampered
ErrHealNotImplemented
ErrHealNoSuchProcess
ErrHealInvalidClientToken
ErrHealMissingBucket
ErrHealAlreadyRunning
ErrHealOverlappingPaths
)
// error code to APIError structure, these fields carry respective
@@ -520,6 +532,12 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "Please reduce your request",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrInvalidPrefixMarker: {
Code: "InvalidPrefixMarker",
Description: "Invalid marker prefix combination",
HTTPStatusCode: http.StatusBadRequest,
},
// FIXME: Actual XML error response also contains the header which missed in list of signed header parameters.
ErrUnsignedHeaders: {
Code: "AccessDenied",
@@ -662,6 +680,11 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed.",
HTTPStatusCode: http.StatusInternalServerError,
},
ErrRequestBodyParse: {
Code: "XMinioRequestBodyParse",
Description: "The request body failed to parse.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrObjectExistsAsDirectory: {
Code: "XMinioObjectExistsAsDirectory",
Description: "Object name already exists as a directory.",
@@ -697,6 +720,11 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "Server not initialized, please try again.",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrMalformedJSON: {
Code: "XMinioMalformedJSON",
Description: "The JSON you provided was not well-formed or did not validate against our published format.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrAdminInvalidAccessKey: {
Code: "XMinioAdminInvalidAccessKey",
Description: "The access key is invalid.",
@@ -712,6 +740,17 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "Configuration update failed because server quorum was not met",
HTTPStatusCode: http.StatusServiceUnavailable,
},
ErrAdminConfigTooLarge: {
Code: "XMinioAdminConfigTooLarge",
Description: fmt.Sprintf("Configuration data provided exceeds the allowed maximum of %d bytes",
maxConfigJSONSize),
HTTPStatusCode: http.StatusBadRequest,
},
ErrAdminConfigBadJSON: {
Code: "XMinioAdminConfigBadJSON",
Description: "JSON configuration provided has objects with duplicate keys",
HTTPStatusCode: http.StatusBadRequest,
},
ErrAdminCredentialsMismatch: {
Code: "XMinioAdminCredentialsMismatch",
Description: "Credentials in config mismatch with server environment variables",
@@ -755,6 +794,36 @@ var errorCodeResponse = map[APIErrorCode]APIError{
Description: "Invalid Request",
HTTPStatusCode: http.StatusBadRequest,
},
ErrHealNotImplemented: {
Code: "XMinioHealNotImplemented",
Description: "This server does not implement heal functionality.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrHealNoSuchProcess: {
Code: "XMinioHealNoSuchProcess",
Description: "No such heal process is running on the server",
HTTPStatusCode: http.StatusBadRequest,
},
ErrHealInvalidClientToken: {
Code: "XMinioHealInvalidClientToken",
Description: "Client token mismatch",
HTTPStatusCode: http.StatusBadRequest,
},
ErrHealMissingBucket: {
Code: "XMinioHealMissingBucket",
Description: "A heal start request with a non-empty object-prefix parameter requires a bucket to be specified.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrHealAlreadyRunning: {
Code: "XMinioHealAlreadyRunning",
Description: "",
HTTPStatusCode: http.StatusBadRequest,
},
ErrHealOverlappingPaths: {
Code: "XMinioHealOverlappingPaths",
Description: "",
HTTPStatusCode: http.StatusBadRequest,
},
// Add your error structure here.
}
+9
View File
@@ -18,6 +18,7 @@ package cmd
import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
"net/http"
@@ -53,6 +54,14 @@ func encodeResponse(response interface{}) []byte {
return bytesBuffer.Bytes()
}
// Encodes the response headers into JSON format.
func encodeResponseJSON(response interface{}) []byte {
var bytesBuffer bytes.Buffer
e := json.NewEncoder(&bytesBuffer)
e.Encode(response)
return bytesBuffer.Bytes()
}
// Write object header
func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, contentRange *httpRange) {
// set common headers
+37 -16
View File
@@ -166,13 +166,12 @@ type ListBucketsResponse struct {
// Upload container for in progress multipart upload
type Upload struct {
Key string
UploadID string `xml:"UploadId"`
Initiator Initiator
Owner Owner
StorageClass string
Initiated string
HealUploadInfo *HealObjectInfo `xml:"HealObjectInfo,omitempty"`
Key string
UploadID string `xml:"UploadId"`
Initiator Initiator
Owner Owner
StorageClass string
Initiated string
}
// CommonPrefix container for prefix response in ListObjectsResponse
@@ -182,9 +181,8 @@ type CommonPrefix struct {
// Bucket container for bucket metadata
type Bucket struct {
Name string
CreationDate string // time string of format "2006-01-02T15:04:05.000Z"
HealBucketInfo *HealBucketInfo `xml:"HealBucketInfo,omitempty"`
Name string
CreationDate string // time string of format "2006-01-02T15:04:05.000Z"
}
// Object container for object metadata
@@ -198,8 +196,7 @@ type Object struct {
Owner Owner
// The class of storage used to store the object.
StorageClass string
HealObjectInfo *HealObjectInfo `xml:"HealObjectInfo,omitempty"`
StorageClass string
}
// CopyObjectResponse container returns ETag and LastModified of the successfully copied object
@@ -308,7 +305,6 @@ func generateListBucketsResponse(buckets []BucketInfo) ListBucketsResponse {
var listbucket = Bucket{}
listbucket.Name = bucket.Name
listbucket.CreationDate = bucket.Created.UTC().Format(timeFormatAMZLong)
listbucket.HealBucketInfo = bucket.HealBucketInfo
listbuckets = append(listbuckets, listbucket)
}
@@ -339,8 +335,6 @@ func generateListObjectsV1Response(bucket, prefix, marker, delimiter string, max
content.Size = object.Size
content.StorageClass = globalMinioDefaultStorageClass
content.Owner = owner
// object.HealObjectInfo is non-empty only when resp is constructed in ListObjectsHeal.
content.HealObjectInfo = object.HealObjectInfo
contents = append(contents, content)
}
// TODO - support EncodingType in xml decoding
@@ -498,7 +492,6 @@ func generateListMultipartUploadsResponse(bucket string, multipartsInfo ListMult
newUpload.UploadID = upload.UploadID
newUpload.Key = upload.Object
newUpload.Initiated = upload.Initiated.UTC().Format(timeFormatAMZLong)
newUpload.HealUploadInfo = upload.HealUploadInfo
listMultipartUploadsResponse.Uploads[index] = newUpload
}
return listMultipartUploadsResponse
@@ -584,3 +577,31 @@ func writeErrorResponseHeadersOnly(w http.ResponseWriter, errorCode APIErrorCode
apiError := getAPIError(errorCode)
writeResponse(w, apiError.HTTPStatusCode, nil, mimeNone)
}
// writeErrorResponseJSON - writes error response in JSON format;
// useful for admin APIs.
func writeErrorResponseJSON(w http.ResponseWriter, errorCode APIErrorCode, reqURL *url.URL) {
apiError := getAPIError(errorCode)
// Generate error response.
errorResponse := getAPIErrorResponse(apiError, reqURL.Path)
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, apiError.HTTPStatusCode, encodedErrorResponse, mimeJSON)
}
// writeCustomErrorResponseJSON - similar to writeErrorResponseJSON,
// but accepts the error message directly (this allows messages to be
// dynamically generated.)
func writeCustomErrorResponseJSON(w http.ResponseWriter, errorCode APIErrorCode,
errBody string, reqURL *url.URL) {
apiError := getAPIError(errorCode)
errorResponse := APIErrorResponse{
Code: apiError.Code,
Message: errBody,
Resource: reqURL.Path,
RequestID: "3L137",
HostID: "3L137",
}
encodedErrorResponse := encodeResponseJSON(errorResponse)
writeResponse(w, apiError.HTTPStatusCode, encodedErrorResponse, mimeJSON)
}
+17 -7
View File
@@ -18,16 +18,12 @@ package cmd
import (
"bytes"
"errors"
"io/ioutil"
"net/http"
"strings"
)
// Verify if the request http Header "x-amz-content-sha256" == "UNSIGNED-PAYLOAD"
func isRequestUnsignedPayload(r *http.Request) bool {
return r.Header.Get("x-amz-content-sha256") == unsignedPayload
}
// Verify if request has JWT.
func isRequestJWT(r *http.Request) bool {
return strings.HasPrefix(r.Header.Get("Authorization"), jwtAlgorithm)
@@ -58,13 +54,14 @@ func isRequestPresignedSignatureV2(r *http.Request) bool {
// Verify if request has AWS Post policy Signature Version '4'.
func isRequestPostPolicySignatureV4(r *http.Request) bool {
return strings.Contains(r.Header.Get("Content-Type"), "multipart/form-data") && r.Method == httpPOST
return strings.Contains(r.Header.Get("Content-Type"), "multipart/form-data") &&
r.Method == http.MethodPost
}
// Verify if the request has AWS Streaming Signature Version '4'. This is only valid for 'PUT' operation.
func isRequestSignStreamingV4(r *http.Request) bool {
return r.Header.Get("x-amz-content-sha256") == streamingContentSHA256 &&
r.Method == httpPUT
r.Method == http.MethodPut
}
// Authorization type.
@@ -105,6 +102,19 @@ func getRequestAuthType(r *http.Request) authType {
return authTypeUnknown
}
// checkAdminRequestAuthType checks whether the request is a valid signature V2 or V4 request.
// It does not accept presigned or JWT or anonymous requests.
func checkAdminRequestAuthType(r *http.Request, region string) APIErrorCode {
s3Err := ErrAccessDenied
if getRequestAuthType(r) == authTypeSigned { // we only support V4 (no presign)
s3Err = isReqAuthenticated(r, region)
}
if s3Err != ErrNone {
errorIf(errors.New(getAPIError(s3Err).Description), "%s", dumpRequest(r))
}
return s3Err
}
func checkRequestAuthType(r *http.Request, bucket, policyAction, region string) APIErrorCode {
reqAuthType := getRequestAuthType(r)
+63 -34
View File
@@ -23,6 +23,7 @@ import (
"net/url"
"os"
"testing"
"time"
"github.com/minio/minio/pkg/auth"
)
@@ -190,40 +191,6 @@ func TestS3SupportedAuthType(t *testing.T) {
}
}
// TestIsRequestUnsignedPayload - Test validates the Unsigned payload detection logic.
func TestIsRequestUnsignedPayload(t *testing.T) {
testCases := []struct {
inputAmzContentHeader string
expectedResult bool
}{
// Test case - 1.
// Test case with "X-Amz-Content-Sha256" header set to empty value.
{"", false},
// Test case - 2.
// Test case with "X-Amz-Content-Sha256" header set to "UNSIGNED-PAYLOAD"
// The payload is flagged as unsigned When "X-Amz-Content-Sha256" header is set to "UNSIGNED-PAYLOAD".
{unsignedPayload, true},
// Test case - 3.
// set to a random value.
{"abcd", false},
}
// creating an input HTTP request.
// Only the headers are relevant for this particular test.
inputReq, err := http.NewRequest("GET", "http://example.com", nil)
if err != nil {
t.Fatalf("Error initializing input HTTP request: %v", err)
}
for i, testCase := range testCases {
inputReq.Header.Set("X-Amz-Content-Sha256", testCase.inputAmzContentHeader)
actualResult := isRequestUnsignedPayload(inputReq)
if testCase.expectedResult != actualResult {
t.Errorf("Test %d: Expected the result to `%v`, but instead got `%v`", i+1, testCase.expectedResult, actualResult)
}
}
}
func TestIsRequestPresignedSignatureV2(t *testing.T) {
testCases := []struct {
inputQueryKey string
@@ -311,6 +278,39 @@ func mustNewSignedRequest(method string, urlStr string, contentLength int64, bod
return req
}
// This is similar to mustNewRequest but additionally the request
// is signed with AWS Signature V2, fails if not able to do so.
func mustNewSignedV2Request(method string, urlStr string, contentLength int64, body io.ReadSeeker, t *testing.T) *http.Request {
req := mustNewRequest(method, urlStr, contentLength, body, t)
cred := globalServerConfig.GetCredential()
if err := signRequestV2(req, cred.AccessKey, cred.SecretKey); err != nil {
t.Fatalf("Unable to inititalized new signed http request %s", err)
}
return req
}
// This is similar to mustNewRequest but additionally the request
// is presigned with AWS Signature V2, fails if not able to do so.
func mustNewPresignedV2Request(method string, urlStr string, contentLength int64, body io.ReadSeeker, t *testing.T) *http.Request {
req := mustNewRequest(method, urlStr, contentLength, body, t)
cred := globalServerConfig.GetCredential()
if err := preSignV2(req, cred.AccessKey, cred.SecretKey, time.Now().Add(10*time.Minute).Unix()); err != nil {
t.Fatalf("Unable to inititalized new signed http request %s", err)
}
return req
}
// This is similar to mustNewRequest but additionally the request
// is presigned with AWS Signature V4, fails if not able to do so.
func mustNewPresignedRequest(method string, urlStr string, contentLength int64, body io.ReadSeeker, t *testing.T) *http.Request {
req := mustNewRequest(method, urlStr, contentLength, body, t)
cred := globalServerConfig.GetCredential()
if err := preSignV4(req, cred.AccessKey, cred.SecretKey, time.Now().Add(10*time.Minute).Unix()); err != nil {
t.Fatalf("Unable to inititalized new signed http request %s", err)
}
return req
}
func mustNewSignedBadMD5Request(method string, urlStr string, contentLength int64, body io.ReadSeeker, t *testing.T) *http.Request {
req := mustNewRequest(method, urlStr, contentLength, body, t)
req.Header.Set("Content-Md5", "YWFhYWFhYWFhYWFhYWFhCg==")
@@ -358,3 +358,32 @@ func TestIsReqAuthenticated(t *testing.T) {
}
}
}
func TestCheckAdminRequestAuthType(t *testing.T) {
path, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatalf("unable initialize config file, %s", err)
}
defer os.RemoveAll(path)
creds, err := auth.CreateCredentials("myuser", "mypassword")
if err != nil {
t.Fatalf("unable create credential, %s", err)
}
globalServerConfig.SetCredential(creds)
testCases := []struct {
Request *http.Request
ErrCode APIErrorCode
}{
{Request: mustNewRequest("GET", "http://127.0.0.1:9000", 0, nil, t), ErrCode: ErrAccessDenied},
{Request: mustNewSignedRequest("GET", "http://127.0.0.1:9000", 0, nil, t), ErrCode: ErrNone},
{Request: mustNewSignedV2Request("GET", "http://127.0.0.1:9000", 0, nil, t), ErrCode: ErrAccessDenied},
{Request: mustNewPresignedV2Request("GET", "http://127.0.0.1:9000", 0, nil, t), ErrCode: ErrAccessDenied},
{Request: mustNewPresignedRequest("GET", "http://127.0.0.1:9000", 0, nil, t), ErrCode: ErrAccessDenied},
}
for i, testCase := range testCases {
if s3Error := checkAdminRequestAuthType(testCase.Request, globalServerConfig.GetRegion()); s3Error != testCase.ErrCode {
t.Errorf("Test %d: Unexpected s3error returned wanted %d, got %d", i, testCase.ErrCode, s3Error)
}
}
}
+139 -13
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016, 2017 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017, 2018 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,7 +17,16 @@
package cmd
import (
"bufio"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/rpc"
"strings"
"sync"
"time"
)
@@ -52,10 +61,11 @@ type authConfig struct {
// AuthRPCClient is a authenticated RPC client which does authentication before doing Call().
type AuthRPCClient struct {
sync.RWMutex // Mutex to lock this object.
rpcClient *RPCClient // Reconnectable RPC client to make any RPC call.
config authConfig // Authentication configuration information.
authToken string // Authentication token.
sync.RWMutex // Mutex to lock this object.
rpcClient *rpc.Client // RPC client to make any RPC call.
config authConfig // Authentication configuration information.
authToken string // Authentication token.
version semVersion // RPC version.
}
// newAuthRPCClient - returns a JWT based authenticated (go) rpc client, which does automatic reconnect.
@@ -73,8 +83,8 @@ func newAuthRPCClient(config authConfig) *AuthRPCClient {
}
return &AuthRPCClient{
rpcClient: newRPCClient(config.serverAddr, config.serviceEndpoint, config.secureConn),
config: config,
config: config,
version: globalRPCAPIVersion,
}
}
@@ -99,38 +109,60 @@ func (authClient *AuthRPCClient) Login() (err error) {
// Attempt to login if not logged in already.
if authClient.authToken == "" {
authClient.authToken, err = authenticateNode(authClient.config.accessKey, authClient.config.secretKey)
var authToken string
authToken, err = authenticateNode(authClient.config.accessKey, authClient.config.secretKey)
if err != nil {
return err
}
// Login to authenticate your token.
var (
loginMethod = authClient.config.serviceName + loginMethodName
loginArgs = LoginRPCArgs{
AuthToken: authClient.authToken,
Version: Version,
AuthToken: authToken,
Version: globalRPCAPIVersion,
RequestTime: UTCNow(),
}
)
if err = authClient.rpcClient.Call(loginMethod, &loginArgs, &LoginRPCReply{}); err != nil {
// Re-dial after we have disconnected or if its a fresh run.
var rpcClient *rpc.Client
rpcClient, err = rpcDial(authClient.config.serverAddr, authClient.config.serviceEndpoint, authClient.config.secureConn)
if err != nil {
return err
}
if err = rpcClient.Call(loginMethod, &loginArgs, &LoginRPCReply{}); err != nil {
// gob doesn't provide any typed errors for us to reflect
// upon, this is the only way to return proper error.
if strings.Contains(err.Error(), "gob: wrong type") {
return errRPCAPIVersionUnsupported
}
return err
}
// Initialize rpc client and auth token after a successful login.
authClient.authToken = authToken
authClient.rpcClient = rpcClient
}
return nil
}
// call makes a RPC call after logs into the server.
func (authClient *AuthRPCClient) call(serviceMethod string, args interface {
SetAuthToken(authToken string)
SetRPCAPIVersion(version semVersion)
}, reply interface{}) (err error) {
if err = authClient.Login(); err != nil {
return err
} // On successful login, execute RPC call.
authClient.RLock()
// Set token before the rpc call.
authClient.RLock()
defer authClient.RUnlock()
args.SetAuthToken(authClient.authToken)
authClient.RUnlock()
args.SetRPCAPIVersion(authClient.version)
// Do an RPC call.
return authClient.rpcClient.Call(serviceMethod, args, reply)
@@ -139,6 +171,7 @@ func (authClient *AuthRPCClient) call(serviceMethod string, args interface {
// Call executes RPC call till success or globalAuthRPCRetryThreshold on ErrShutdown.
func (authClient *AuthRPCClient) Call(serviceMethod string, args interface {
SetAuthToken(authToken string)
SetRPCAPIVersion(version semVersion)
}, reply interface{}) (err error) {
// Done channel is used to close any lingering retry routine, as soon
@@ -159,6 +192,11 @@ func (authClient *AuthRPCClient) Call(serviceMethod string, args interface {
}
}
}
// gob doesn't provide any typed errors for us to reflect
// upon, this is the only way to return proper error.
if err != nil && strings.Contains(err.Error(), "gob: wrong type") {
err = errRPCAPIVersionUnsupported
}
break
}
return err
@@ -169,6 +207,10 @@ func (authClient *AuthRPCClient) Close() error {
authClient.Lock()
defer authClient.Unlock()
if authClient.rpcClient == nil {
return nil
}
authClient.authToken = ""
return authClient.rpcClient.Close()
}
@@ -182,3 +224,87 @@ func (authClient *AuthRPCClient) ServerAddr() string {
func (authClient *AuthRPCClient) ServiceEndpoint() string {
return authClient.config.serviceEndpoint
}
// default Dial timeout for RPC connections.
const defaultDialTimeout = 3 * time.Second
// Connect success message required from rpc server.
const connectSuccessMessage = "200 Connected to Go RPC"
// dial tries to establish a connection to serverAddr in a safe manner.
// If there is a valid rpc.Cliemt, it returns that else creates a new one.
func rpcDial(serverAddr, serviceEndpoint string, secureConn bool) (netRPCClient *rpc.Client, err error) {
if serverAddr == "" || serviceEndpoint == "" {
return nil, errInvalidArgument
}
d := &net.Dialer{
Timeout: defaultDialTimeout,
}
var conn net.Conn
if secureConn {
var hostname string
if hostname, _, err = net.SplitHostPort(serverAddr); err != nil {
return nil, &net.OpError{
Op: "dial-http",
Net: serverAddr + serviceEndpoint,
Addr: nil,
Err: fmt.Errorf("Unable to parse server address <%s>: %s", serverAddr, err),
}
}
// ServerName in tls.Config needs to be specified to support SNI certificates.
conn, err = tls.DialWithDialer(d, "tcp", serverAddr, &tls.Config{
ServerName: hostname,
RootCAs: globalRootCAs,
})
} else {
conn, err = d.Dial("tcp", serverAddr)
}
if err != nil {
// Print RPC connection errors that are worthy to display in log.
switch err.(type) {
case x509.HostnameError:
errorIf(err, "Unable to establish secure connection to %s", serverAddr)
}
return nil, &net.OpError{
Op: "dial-http",
Net: serverAddr + serviceEndpoint,
Addr: nil,
Err: err,
}
}
// Check for network errors writing over the dialed conn.
if _, err = io.WriteString(conn, "CONNECT "+serviceEndpoint+" HTTP/1.0\n\n"); err != nil {
conn.Close()
return nil, &net.OpError{
Op: "dial-http",
Net: serverAddr + serviceEndpoint,
Addr: nil,
Err: err,
}
}
// Attempt to read the HTTP response for the HTTP method CONNECT, upon
// success return the RPC connection instance.
resp, err := http.ReadResponse(bufio.NewReader(conn), &http.Request{
Method: http.MethodConnect,
})
if err != nil {
conn.Close()
return nil, &net.OpError{
Op: "dial-http",
Net: serverAddr + serviceEndpoint,
Addr: nil,
Err: err,
}
}
if resp.Status != connectSuccessMessage {
conn.Close()
return nil, errors.New("unexpected HTTP response: " + resp.Status)
}
// Initialize rpc client.
return rpc.NewClient(conn), nil
}
+84 -1
View File
@@ -16,7 +16,12 @@
package cmd
import "testing"
import (
"crypto/x509"
"os"
"path"
"testing"
)
// Tests authorized RPC client.
func TestAuthRPCClient(t *testing.T) {
@@ -53,3 +58,81 @@ func TestAuthRPCClient(t *testing.T) {
t.Fatalf("Unexpected node value %s, but expected %s", authRPC.ServiceEndpoint(), authCfg.serviceEndpoint)
}
}
// Test rpc dial test.
func TestRPCDial(t *testing.T) {
prevRootCAs := globalRootCAs
defer func() {
globalRootCAs = prevRootCAs
}()
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(rootPath)
testServer := StartTestServer(t, "")
defer testServer.Stop()
cert, key, err := generateTLSCertKey("127.0.0.1")
if err != nil {
t.Fatal(err)
}
// Set global root CAs.
globalRootCAs = x509.NewCertPool()
globalRootCAs.AppendCertsFromPEM(cert)
testServerTLS := StartTestTLSServer(t, "", cert, key)
defer testServerTLS.Stop()
adminEndpoint := path.Join(minioReservedBucketPath, adminPath)
testCases := []struct {
serverAddr string
serverEndpoint string
success bool
secure bool
}{
// Empty server addr should fail.
{
serverAddr: "",
serverEndpoint: adminEndpoint,
success: false,
},
// Unexpected server addr should fail.
{
serverAddr: "example.com",
serverEndpoint: adminEndpoint,
success: false,
},
// Server addr connects but fails for CONNECT call.
{
serverAddr: "example.com:80",
serverEndpoint: "/",
success: false,
},
// Successful connecting to insecure RPC server.
{
serverAddr: testServer.Server.Listener.Addr().String(),
serverEndpoint: adminEndpoint,
success: true,
},
// Successful connecting to secure RPC server.
{
serverAddr: testServerTLS.Server.Listener.Addr().String(),
serverEndpoint: adminEndpoint,
success: true,
secure: true,
},
}
for i, testCase := range testCases {
_, err = rpcDial(testCase.serverAddr, testCase.serverEndpoint, testCase.secure)
if err != nil && testCase.success {
t.Errorf("Test %d: Expected success but found failure instead %s", i+1, err)
}
if err == nil && !testCase.success {
t.Errorf("Test %d: Expected failure but found success instead", i+1)
}
}
}
+5 -5
View File
@@ -43,7 +43,7 @@ func TestLogin(t *testing.T) {
{
args: LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
},
skewTime: 0,
expectedErr: nil,
@@ -52,16 +52,16 @@ func TestLogin(t *testing.T) {
{
args: LoginRPCArgs{
AuthToken: token,
Version: "INVALID-" + Version,
Version: semVersion{2, 0, 0},
},
skewTime: 0,
expectedErr: errServerVersionMismatch,
expectedErr: errRPCAPIVersionUnsupported,
},
// Valid username, password and version, not request time
{
args: LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
},
skewTime: 20 * time.Minute,
expectedErr: errServerTimeMismatch,
@@ -70,7 +70,7 @@ func TestLogin(t *testing.T) {
{
args: LoginRPCArgs{
AuthToken: "",
Version: Version,
Version: globalRPCAPIVersion,
},
skewTime: 0,
expectedErr: errAuthentication,
+2 -2
View File
@@ -239,7 +239,7 @@ func runGetObjectBenchmark(b *testing.B, obj ObjectLayer, objSize int) {
b.ResetTimer()
for i := 0; i < b.N; i++ {
var buffer = new(bytes.Buffer)
err = obj.GetObject(bucket, "object"+strconv.Itoa(i%10), 0, int64(objSize), buffer)
err = obj.GetObject(bucket, "object"+strconv.Itoa(i%10), 0, int64(objSize), buffer, "")
if err != nil {
b.Error(err)
}
@@ -402,7 +402,7 @@ func runGetObjectBenchmarkParallel(b *testing.B, obj ObjectLayer, objSize int) {
b.RunParallel(func(pb *testing.PB) {
i := 0
for pb.Next() {
err = obj.GetObject(bucket, "object"+strconv.Itoa(i), 0, int64(objSize), ioutil.Discard)
err = obj.GetObject(bucket, "object"+strconv.Itoa(i), 0, int64(objSize), ioutil.Discard, "")
if err != nil {
b.Error(err)
}
+4
View File
@@ -50,6 +50,10 @@ func (br *browserPeerAPIHandlers) SetAuthPeer(args SetAuthPeerArgs, reply *AuthR
return fmt.Errorf("Invalid credential passed")
}
// Acquire lock before updating global configuration.
globalServerConfigMu.Lock()
defer globalServerConfigMu.Unlock()
// Update credentials in memory
prevCred := globalServerConfig.SetCredential(args.Creds)
+11 -21
View File
@@ -70,11 +70,13 @@ func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
// Validate for invalid token.
args := SetAuthPeerArgs{Creds: creds}
args.AuthToken = "garbage"
rclient := newRPCClient(s.testAuthConf.serverAddr, s.testAuthConf.serviceEndpoint, false)
rclient := newAuthRPCClient(s.testAuthConf)
defer rclient.Close()
err = rclient.Call("BrowserPeer.SetAuthPeer", &args, &AuthRPCReply{})
if err != nil {
if err = rclient.Login(); err != nil {
t.Fatal(err)
}
rclient.authToken = "garbage"
if err = rclient.Call("BrowserPeer.SetAuthPeer", &args, &AuthRPCReply{}); err != nil {
if err.Error() != errInvalidToken.Error() {
t.Fatal(err)
}
@@ -90,20 +92,14 @@ func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
}
// Validate for failure in login handler with previous credentials.
rclient = newRPCClient(s.testAuthConf.serverAddr, s.testAuthConf.serviceEndpoint, false)
rclient = newAuthRPCClient(s.testAuthConf)
defer rclient.Close()
token, err := authenticateNode(creds.AccessKey, creds.SecretKey)
if err != nil {
t.Fatal(err)
}
rargs := &LoginRPCArgs{
AuthToken: token,
Version: Version,
RequestTime: UTCNow(),
}
rreply := &LoginRPCReply{}
err = rclient.Call("BrowserPeer"+loginMethodName, rargs, rreply)
if err != nil {
rclient.authToken = token
if err = rclient.Login(); err != nil {
if err.Error() != errInvalidAccessKeyID.Error() {
t.Fatal(err)
}
@@ -113,14 +109,8 @@ func (s *TestRPCBrowserPeerSuite) testBrowserPeerRPC(t *testing.T) {
if err != nil {
t.Fatal(err)
}
// Validate for success in loing handled with valid credetnails.
rargs = &LoginRPCArgs{
AuthToken: token,
Version: Version,
RequestTime: UTCNow(),
}
rreply = &LoginRPCReply{}
if err = rclient.Call("BrowserPeer"+loginMethodName, rargs, rreply); err != nil {
rclient.authToken = token
if err = rclient.Login(); err != nil {
t.Fatal(err)
}
}
+17 -3
View File
@@ -44,7 +44,7 @@ func validateListObjectsArgs(prefix, marker, delimiter string, maxKeys int) APIE
if marker != "" {
// Marker not common with prefix is not implemented.
if !hasPrefix(marker, prefix) {
return ErrNotImplemented
return ErrInvalidPrefixMarker
}
}
// Success.
@@ -107,15 +107,23 @@ func (api objectAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *http
// Validate the query params before beginning to serve the request.
// fetch-owner is not validated since it is a boolean
if s3Error := validateListObjectsArgs(prefix, marker, delimiter, maxKeys); s3Error != ErrNone {
// return empty response if invalid marker
//TODO: avoid this pattern when moving to errors package
if s3Error == ErrInvalidPrefixMarker {
listObjectsInfo := ListObjectsInfo{}
response := generateListObjectsV2Response(bucket, prefix, token, marker, startAfter, delimiter, fetchOwner, listObjectsInfo.IsTruncated, maxKeys, listObjectsInfo.Objects, listObjectsInfo.Prefixes)
writeSuccessResponseXML(w, encodeResponse(response))
return
}
writeErrorResponse(w, s3Error, r.URL)
return
}
// Inititate a list objects operation based on the input params.
// On success would return back ListObjectsInfo object to be
// marshalled into S3 compatible XML header.
listObjectsInfo, err := objectAPI.ListObjects(bucket, prefix, marker, delimiter, maxKeys)
if err != nil {
errorIf(err, "Unable to list objects.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -152,6 +160,13 @@ func (api objectAPIHandlers) ListObjectsV1Handler(w http.ResponseWriter, r *http
// Validate all the query params before beginning to serve the request.
if s3Error := validateListObjectsArgs(prefix, marker, delimiter, maxKeys); s3Error != ErrNone {
// return empty response if invalid marker
//TODO: avoid this pattern when moving to errors package
if s3Error == ErrInvalidPrefixMarker {
response := generateListObjectsV1Response(bucket, prefix, marker, delimiter, maxKeys, ListObjectsInfo{})
writeSuccessResponseXML(w, encodeResponse(response))
return
}
writeErrorResponse(w, s3Error, r.URL)
return
}
@@ -161,7 +176,6 @@ func (api objectAPIHandlers) ListObjectsV1Handler(w http.ResponseWriter, r *http
// marshalled into S3 compatible XML header.
listObjectsInfo, err := objectAPI.ListObjects(bucket, prefix, marker, delimiter, maxKeys)
if err != nil {
errorIf(err, "Unable to list objects.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
+3 -34
View File
@@ -128,7 +128,6 @@ func (api objectAPIHandlers) GetBucketLocationHandler(w http.ResponseWriter, r *
}
if _, err := objectAPI.GetBucketInfo(bucket); err != nil {
errorIf(err, "Unable to fetch bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -185,7 +184,6 @@ func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter,
listMultipartsInfo, err := objectAPI.ListMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
if err != nil {
errorIf(err, "Unable to list multipart uploads.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -221,7 +219,6 @@ func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.R
// Invoke the list buckets.
bucketsInfo, err := objectAPI.ListBuckets()
if err != nil {
errorIf(err, "Unable to list buckets.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -304,16 +301,9 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
}
return
}
objectLock := globalNSMutex.NewNSLock(bucket, obj.ObjectName)
if timedOutErr := objectLock.GetLock(globalObjectTimeout); timedOutErr != nil {
dErrs[i] = timedOutErr
} else {
defer objectLock.Unlock()
dErr := objectAPI.DeleteObject(bucket, obj.ObjectName)
if dErr != nil {
dErrs[i] = dErr
}
dErr := objectAPI.DeleteObject(bucket, obj.ObjectName)
if dErr != nil {
dErrs[i] = dErr
}
}(index, object)
}
@@ -335,7 +325,6 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
deletedObjects = append(deletedObjects, object)
continue
}
errorIf(err, "Unable to delete object. %s", object.ObjectName)
// Error during delete should be collected separately.
deleteErrors = append(deleteErrors, DeleteError{
Code: errorCodeResponse[toAPIErrorCode(err)].Code,
@@ -408,17 +397,9 @@ func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Req
return
}
bucketLock := globalNSMutex.NewNSLock(bucket, "")
if bucketLock.GetLock(globalObjectTimeout) != nil {
writeErrorResponse(w, ErrOperationTimedOut, r.URL)
return
}
defer bucketLock.Unlock()
// Proceed to creating a bucket.
err := objectAPI.MakeBucketWithLocation(bucket, "")
if err != nil {
errorIf(err, "Unable to create a bucket.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -546,13 +527,11 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
lengthRange := postPolicyForm.Conditions.ContentLengthRange
if lengthRange.Valid {
if fileSize < lengthRange.Min {
errorIf(err, "Unable to create object.")
writeErrorResponse(w, toAPIErrorCode(errDataTooSmall), r.URL)
return
}
if fileSize > lengthRange.Max || isMaxObjectSize(fileSize) {
errorIf(err, "Unable to create object.")
writeErrorResponse(w, toAPIErrorCode(errDataTooLarge), r.URL)
return
}
@@ -566,13 +545,6 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
return
}
objectLock := globalNSMutex.NewNSLock(bucket, object)
if objectLock.GetLock(globalObjectTimeout) != nil {
writeErrorResponse(w, ErrOperationTimedOut, r.URL)
return
}
defer objectLock.Unlock()
hashReader, err := hash.NewReader(fileBody, fileSize, "", "")
if err != nil {
errorIf(err, "Unable to initialize hashReader.")
@@ -582,7 +554,6 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
objInfo, err := objectAPI.PutObject(bucket, object, hashReader, metadata)
if err != nil {
errorIf(err, "Unable to create object.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -662,7 +633,6 @@ func (api objectAPIHandlers) HeadBucketHandler(w http.ResponseWriter, r *http.Re
defer bucketLock.RUnlock()
if _, err := objectAPI.GetBucketInfo(bucket); err != nil {
errorIf(err, "Unable to fetch bucket info.")
writeErrorResponseHeadersOnly(w, toAPIErrorCode(err))
return
}
@@ -696,7 +666,6 @@ func (api objectAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http.
// Attempt to delete bucket.
if err := objectAPI.DeleteBucket(bucket); err != nil {
errorIf(err, "Unable to delete a bucket.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
-3
View File
@@ -59,7 +59,6 @@ func (api objectAPIHandlers) GetBucketNotificationHandler(w http.ResponseWriter,
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -113,7 +112,6 @@ func (api objectAPIHandlers) PutBucketNotificationHandler(w http.ResponseWriter,
_, err := objectAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -328,7 +326,6 @@ func (api objectAPIHandlers) ListenBucketNotificationHandler(w http.ResponseWrit
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to get bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
-4
View File
@@ -236,7 +236,6 @@ func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *ht
// Before proceeding validate if bucket exists.
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -295,7 +294,6 @@ func (api objectAPIHandlers) DeleteBucketPolicyHandler(w http.ResponseWriter, r
// Before proceeding validate if bucket exists.
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -336,7 +334,6 @@ func (api objectAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *ht
// Before proceeding validate if bucket exists.
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -344,7 +341,6 @@ func (api objectAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *ht
// Read bucket access policy.
policy, err := readBucketPolicy(bucket, objAPI)
if err != nil {
errorIf(err, "Unable to read bucket policy.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
+1 -24
View File
@@ -88,7 +88,6 @@ func loadAllBucketPolicies(objAPI ObjectLayer) (policies map[string]policy.Bucke
// List buckets to proceed loading all notification configuration.
buckets, err := objAPI.ListBuckets()
if err != nil {
errorIf(err, "Unable to list buckets.")
return nil, errors.Cause(err)
}
@@ -149,15 +148,8 @@ func initBucketPolicies(objAPI ObjectLayer) error {
func readBucketPolicyJSON(bucket string, objAPI ObjectLayer) (bucketPolicyReader io.Reader, err error) {
policyPath := pathJoin(bucketConfigPrefix, bucket, bucketPolicyConfig)
// Acquire a read lock on policy config before reading.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, policyPath)
if err = objLock.GetRLock(globalOperationTimeout); err != nil {
return nil, err
}
defer objLock.RUnlock()
var buffer bytes.Buffer
err = objAPI.GetObject(minioMetaBucket, policyPath, 0, -1, &buffer)
err = objAPI.GetObject(minioMetaBucket, policyPath, 0, -1, &buffer, "")
if err != nil {
if isErrObjectNotFound(err) || isErrIncompleteBody(err) {
return nil, BucketPolicyNotFound{Bucket: bucket}
@@ -191,15 +183,8 @@ func readBucketPolicy(bucket string, objAPI ObjectLayer) (policy.BucketAccessPol
// if no policies are found.
func removeBucketPolicy(bucket string, objAPI ObjectLayer) error {
policyPath := pathJoin(bucketConfigPrefix, bucket, bucketPolicyConfig)
// Acquire a write lock on policy config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, policyPath)
if err := objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
defer objLock.Unlock()
err := objAPI.DeleteObject(minioMetaBucket, policyPath)
if err != nil {
errorIf(err, "Unable to remove bucket-policy on bucket %s.", bucket)
err = errors.Cause(err)
if _, ok := err.(ObjectNotFound); ok {
return BucketPolicyNotFound{Bucket: bucket}
@@ -217,13 +202,6 @@ func writeBucketPolicy(bucket string, objAPI ObjectLayer, bpy policy.BucketAcces
return err
}
policyPath := pathJoin(bucketConfigPrefix, bucket, bucketPolicyConfig)
// Acquire a write lock on policy config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, policyPath)
if err = objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
defer objLock.Unlock()
hashReader, err := hash.NewReader(bytes.NewReader(buf), int64(len(buf)), "", getSHA256Hash(buf))
if err != nil {
errorIf(err, "Unable to set policy for the bucket %s", bucket)
@@ -242,7 +220,6 @@ func parseAndPersistBucketPolicy(bucket string, policyBytes []byte, objAPI Objec
var bktPolicy policy.BucketAccessPolicy
err := parseBucketPolicy(bytes.NewReader(policyBytes), &bktPolicy)
if err != nil {
errorIf(err, "Unable to parse bucket policy.")
return ErrInvalidPolicyDocument
}
+37 -1
View File
@@ -22,9 +22,15 @@ import (
"encoding/pem"
"fmt"
"io/ioutil"
"os"
"path/filepath"
)
// TLSPrivateKeyPassword is the environment variable which contains the password used
// to decrypt the TLS private key. It must be set if the TLS private key is
// password protected.
const TLSPrivateKeyPassword = "MINIO_CERT_PASSWD"
func parsePublicCertFile(certFile string) (x509Certs []*x509.Certificate, err error) {
// Read certificate file.
var data []byte
@@ -90,6 +96,36 @@ func getRootCAs(certsCAsDir string) (*x509.CertPool, error) {
return rootCAs, nil
}
// load an X509 key pair (private key , certificate) from the provided
// paths. The private key may be encrypted and is decrypted using the
// ENV_VAR: MINIO_CERT_PASSWD.
func loadX509KeyPair(certFile, keyFile string) (tls.Certificate, error) {
certPEMBlock, err := ioutil.ReadFile(certFile)
if err != nil {
return tls.Certificate{}, fmt.Errorf("TLS: failed to read cert file: %v", err)
}
keyPEMBlock, err := ioutil.ReadFile(keyFile)
if err != nil {
return tls.Certificate{}, fmt.Errorf("TLS: failed to read private key: %v", err)
}
key, rest := pem.Decode(keyPEMBlock)
if len(rest) > 0 {
return tls.Certificate{}, fmt.Errorf("TLS: private key contains additional data")
}
if x509.IsEncryptedPEMBlock(key) {
password, ok := os.LookupEnv(TLSPrivateKeyPassword)
if !ok {
return tls.Certificate{}, fmt.Errorf("TLS: private key is encrypted but no password is present - set env var: %s", TLSPrivateKeyPassword)
}
decryptedKey, decErr := x509.DecryptPEMBlock(key, []byte(password))
if decErr != nil {
return tls.Certificate{}, fmt.Errorf("TLS: failed to decrypt private key: %v", decErr)
}
keyPEMBlock = pem.EncodeToMemory(&pem.Block{Type: key.Type, Bytes: decryptedKey})
}
return tls.X509KeyPair(certPEMBlock, keyPEMBlock)
}
func getSSLConfig() (x509Certs []*x509.Certificate, rootCAs *x509.CertPool, tlsCert *tls.Certificate, secureConn bool, err error) {
if !(isFile(getPublicCertFile()) && isFile(getPrivateKeyFile())) {
return nil, nil, nil, false, nil
@@ -100,7 +136,7 @@ func getSSLConfig() (x509Certs []*x509.Certificate, rootCAs *x509.CertPool, tlsC
}
var cert tls.Certificate
if cert, err = tls.LoadX509KeyPair(getPublicCertFile(), getPrivateKeyFile()); err != nil {
if cert, err = loadX509KeyPair(getPublicCertFile(), getPrivateKeyFile()); err != nil {
return nil, nil, nil, false, err
}
+249
View File
@@ -257,3 +257,252 @@ func TestGetRootCAs(t *testing.T) {
}
}
}
func TestLoadX509KeyPair(t *testing.T) {
for i, testCase := range loadX509KeyPairTests {
privateKey, err := createTempFile("private.key", testCase.privateKey)
if err != nil {
t.Fatalf("Test %d: failed to create tmp private key file: %v", i, err)
}
certificate, err := createTempFile("public.crt", testCase.certificate)
if err != nil {
os.Remove(privateKey)
t.Fatalf("Test %d: failed to create tmp certificate file: %v", i, err)
}
os.Unsetenv(TLSPrivateKeyPassword)
if testCase.password != "" {
os.Setenv(TLSPrivateKeyPassword, testCase.password)
}
_, err = loadX509KeyPair(certificate, privateKey)
if err != nil && !testCase.shouldFail {
t.Errorf("Test %d: test should succeed but it failed: %v", i, err)
}
if err == nil && testCase.shouldFail {
t.Errorf("Test %d: test should fail but it succeed", i)
}
os.Remove(privateKey)
os.Remove(certificate)
}
}
var loadX509KeyPairTests = []struct {
password string
privateKey, certificate string
shouldFail bool
}{
{
password: "foobar",
privateKey: `-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,CC483BF11678C35F9F02A1AD85DAE285
nMDFd+Qxk1f+S7LwMitmMofNXYNbCY4L1QEqPOOx5wnjNF1wSxmEkL7+h8W4Y/vb
AQt/7TCcUSuSqEMl45nUIcCbhBos5wz+ShvFiez3qKwmR5HSURvqyN6PIJeAbU+h
uw/cvAQsCH1Cq+gYkDJqjrizPhGqg7mSkqyeST3PbOl+ZXc0wynIjA34JSwO3c5j
cF7XKHETtNGj1+AiLruX4wYZAJwQnK375fCoNVMO992zC6K83d8kvGMUgmJjkiIj
q3s4ymFGfoo0S/XNDQXgE5A5QjAKRKUyW2i7pHIIhTyOpeJQeFHDi2/zaZRxoCog
lD2/HKLi5xJtRelZaaGyEJ20c05VzaSZ+EtRIN33foNdyQQL6iAUU3hJ6JlcmRIB
bRfX4XPH1w9UfFU5ZKwUciCoDcL65bsyv/y56ItljBp7Ok+UUKl0H4myFNOSfsuU
IIj4neslnAvwQ8SN4XUpug+7pGF+2m/5UDwRzSUN1H2RfgWN95kqR+tYqCq/E+KO
i0svzFrljSHswsFoPBqKngI7hHwc9QTt5q4frXwj9I4F6HHrTKZnC5M4ef26sbJ1
r7JRmkt0h/GfcS355b0uoBTtF1R8tSJo85Zh47wE+ucdjEvy9/pjnzKqIoJo9bNZ
ri+ue7GhH5EUca1Kd10bH8FqTF+8AHh4yW6xMxSkSgFGp7KtraAVpdp+6kosymqh
dz9VMjA8i28btfkS2isRaCpyumaFYJ3DJMFYhmeyt6gqYovmRLX0qrBf8nrkFTAA
ZmykWsc8ErsCudxlDmKVemuyFL7jtm9IRPq+Jh+IrmixLJFx8PKkNAM6g+A8irx8
piw+yhRsVy5Jk2QeIqvbpxN6BfCNcix4sWkusiCJrAqQFuSm26Mhh53Ig1DXG4d3
6QY1T8tW80Q6JHUtDR+iOPqW6EmrNiEopzirvhGv9FicXZ0Lo2yKJueeeihWhFLL
GmlnCjWVMO4hoo8lWCHv95JkPxGMcecCacKKUbHlXzCGyw3+eeTEHMWMEhziLeBy
HZJ1/GReI3Sx7XlUCkG4468Yz3PpmbNIk/U5XKE7TGuxKmfcWQpu022iF/9DrKTz
KVhKimCBXJX345bCFe1rN2z5CV6sv87FkMs5Y+OjPw6qYFZPVKO2TdUUBcpXbQMg
UW+Kuaax9W7214Stlil727MjRCiH1+0yODg4nWj4pTSocA5R3pn5cwqrjMu97OmL
ESx4DHmy4keeSy3+AIAehCZlwgeLb70/xCSRhJMIMS9Q6bz8CPkEWN8bBZt95oeo
37LqZ7lNmq61fs1x1tq0VUnI9HwLFEnsiubp6RG0Yu8l/uImjjjXa/ytW2GXrfUi
zM22dOntu6u23iBxRBJRWdFTVUz7qrdu+PHavr+Y7TbCeiBwiypmz5llf823UIVx
btamI6ziAq2gKZhObIhut7sjaLkAyTLlNVkNN1WNaplAXpW25UFVk93MHbvZ27bx
9iLGs/qB2kDTUjffSQoHTLY1GoLxv83RgVspUGQjslztEEpWfYvGfVLcgYLv933B
aRW9BRoNZ0czKx7Lhuwjreyb5IcWDarhC8q29ZkkWsQQonaPb0kTEFJul80Yqk0k
-----END RSA PRIVATE KEY-----`,
certificate: `-----BEGIN CERTIFICATE-----
MIIDiTCCAnGgAwIBAgIJAK5m5S7EE46kMA0GCSqGSIb3DQEBCwUAMFsxCzAJBgNV
BAYTAlVTMQ4wDAYDVQQIDAVzdGF0ZTERMA8GA1UEBwwIbG9jYXRpb24xFTATBgNV
BAoMDG9yZ2FuaXphdGlvbjESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTE3MTIxODE4
MDUyOFoXDTI3MTIxNjE4MDUyOFowWzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBXN0
YXRlMREwDwYDVQQHDAhsb2NhdGlvbjEVMBMGA1UECgwMb3JnYW5pemF0aW9uMRIw
EAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQDPJfYY5Dhsntrqwyu7ZgKM/zrlKEjCwGHhWJBdZdeZCHQlY8ISrtDxxp2XMmI6
HsszalEhNF9fk3vSXWclTuomG03fgGzP4R6QpcwGUCxhRF1J+0b64Yi8pw2uEGsR
GuMwLhGorcWalNoihgHc0BQ4vO8aaTNTX7iD06olesP6vGNu/S8h0VomE+0v9qYc
VF66Zaiv/6OmxAtDpElJjVd0mY7G85BlDlFrVwzd7zhRiuJZ4iDg749Xt9GuuKla
Dvr14glHhP4dQgUbhluJmIHMdx2ZPjk+5FxaDK6I9IUpxczFDe4agDE6lKzU1eLd
cCXRWFOf6q9lTB1hUZfmWfTxAgMBAAGjUDBOMB0GA1UdDgQWBBTQh7lDTq+8salD
0HBNILochiiNaDAfBgNVHSMEGDAWgBTQh7lDTq+8salD0HBNILochiiNaDAMBgNV
HRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQAqi9LycxcXKNSDXaPkCKvw7RQy
iMBDGm1kIY++p3tzbUGuaeu85TsswKnqd50AullEU+aQxRRJGfR8eSKzQJMBXLMQ
b4ptYCc5OrZtRHT8NaZ/df2tc6I88kN8dBu6ybcNGsevXA/iNX3kKLW7naxdr5jj
KUudWSuqDCjCmQa5bYb9H6DreLH2lUItSWBa/YmeZ3VSezDCd+XYO53QKwZVj8Jb
bulZmoo7e7HO1qecEzWKL10UYyEbG3UDPtw+NZc142ZYeEhXQ0dsstGAO5hf3hEl
kQyKGUTpDbKLuyYMFsoH73YLjBqNe+UEhPwE+FWpcky1Sp9RTx/oMLpiZaPR
-----END CERTIFICATE-----`,
shouldFail: false,
},
{
password: "password",
privateKey: `-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,CC483BF11678C35F9F02A1AD85DAE285
nMDFd+Qxk1f+S7LwMitmMofNXYNbCY4L1QEqPOOx5wnjNF1wSxmEkL7+h8W4Y/vb
AQt/7TCcUSuSqEMl45nUIcCbhBos5wz+ShvFiez3qKwmR5HSURvqyN6PIJeAbU+h
uw/cvAQsCH1Cq+gYkDJqjrizPhGqg7mSkqyeST3PbOl+ZXc0wynIjA34JSwO3c5j
cF7XKHETtNGj1+AiLruX4wYZAJwQnK375fCoNVMO992zC6K83d8kvGMUgmJjkiIj
q3s4ymFGfoo0S/XNDQXgE5A5QjAKRKUyW2i7pHIIhTyOpeJQeFHDi2/zaZRxoCog
lD2/HKLi5xJtRelZaaGyEJ20c05VzaSZ+EtRIN33foNdyQQL6iAUU3hJ6JlcmRIB
bRfX4XPH1w9UfFU5ZKwUciCoDcL65bsyv/y56ItljBp7Ok+UUKl0H4myFNOSfsuU
IIj4neslnAvwQ8SN4XUpug+7pGF+2m/5UDwRzSUN1H2RfgWN95kqR+tYqCq/E+KO
i0svzFrljSHswsFoPBqKngI7hHwc9QTt5q4frXwj9I4F6HHrTKZnC5M4ef26sbJ1
r7JRmkt0h/GfcS355b0uoBTtF1R8tSJo85Zh47wE+ucdjEvy9/pjnzKqIoJo9bNZ
ri+ue7GhH5EUca1Kd10bH8FqTF+8AHh4yW6xMxSkSgFGp7KtraAVpdp+6kosymqh
dz9VMjA8i28btfkS2isRaCpyumaFYJ3DJMFYhmeyt6gqYovmRLX0qrBf8nrkFTAA
ZmykWsc8ErsCudxlDmKVemuyFL7jtm9IRPq+Jh+IrmixLJFx8PKkNAM6g+A8irx8
piw+yhRsVy5Jk2QeIqvbpxN6BfCNcix4sWkusiCJrAqQFuSm26Mhh53Ig1DXG4d3
6QY1T8tW80Q6JHUtDR+iOPqW6EmrNiEopzirvhGv9FicXZ0Lo2yKJueeeihWhFLL
GmlnCjWVMO4hoo8lWCHv95JkPxGMcecCacKKUbHlXzCGyw3+eeTEHMWMEhziLeBy
HZJ1/GReI3Sx7XlUCkG4468Yz3PpmbNIk/U5XKE7TGuxKmfcWQpu022iF/9DrKTz
KVhKimCBXJX345bCFe1rN2z5CV6sv87FkMs5Y+OjPw6qYFZPVKO2TdUUBcpXbQMg
UW+Kuaax9W7214Stlil727MjRCiH1+0yODg4nWj4pTSocA5R3pn5cwqrjMu97OmL
ESx4DHmy4keeSy3+AIAehCZlwgeLb70/xCSRhJMIMS9Q6bz8CPkEWN8bBZt95oeo
37LqZ7lNmq61fs1x1tq0VUnI9HwLFEnsiubp6RG0Yu8l/uImjjjXa/ytW2GXrfUi
zM22dOntu6u23iBxRBJRWdFTVUz7qrdu+PHavr+Y7TbCeiBwiypmz5llf823UIVx
btamI6ziAq2gKZhObIhut7sjaLkAyTLlNVkNN1WNaplAXpW25UFVk93MHbvZ27bx
9iLGs/qB2kDTUjffSQoHTLY1GoLxv83RgVspUGQjslztEEpWfYvGfVLcgYLv933B
aRW9BRoNZ0czKx7Lhuwjreyb5IcWDarhC8q29ZkkWsQQonaPb0kTEFJul80Yqk0k
-----END RSA PRIVATE KEY-----`,
certificate: `-----BEGIN CERTIFICATE-----
MIIDiTCCAnGgAwIBAgIJAK5m5S7EE46kMA0GCSqGSIb3DQEBCwUAMFsxCzAJBgNV
BAYTAlVTMQ4wDAYDVQQIDAVzdGF0ZTERMA8GA1UEBwwIbG9jYXRpb24xFTATBgNV
BAoMDG9yZ2FuaXphdGlvbjESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTE3MTIxODE4
MDUyOFoXDTI3MTIxNjE4MDUyOFowWzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBXN0
YXRlMREwDwYDVQQHDAhsb2NhdGlvbjEVMBMGA1UECgwMb3JnYW5pemF0aW9uMRIw
EAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQDPJfYY5Dhsntrqwyu7ZgKM/zrlKEjCwGHhWJBdZdeZCHQlY8ISrtDxxp2XMmI6
HsszalEhNF9fk3vSXWclTuomG03fgGzP4R6QpcwGUCxhRF1J+0b64Yi8pw2uEGsR
GuMwLhGorcWalNoihgHc0BQ4vO8aaTNTX7iD06olesP6vGNu/S8h0VomE+0v9qYc
VF66Zaiv/6OmxAtDpElJjVd0mY7G85BlDlFrVwzd7zhRiuJZ4iDg749Xt9GuuKla
Dvr14glHhP4dQgUbhluJmIHMdx2ZPjk+5FxaDK6I9IUpxczFDe4agDE6lKzU1eLd
cCXRWFOf6q9lTB1hUZfmWfTxAgMBAAGjUDBOMB0GA1UdDgQWBBTQh7lDTq+8salD
0HBNILochiiNaDAfBgNVHSMEGDAWgBTQh7lDTq+8salD0HBNILochiiNaDAMBgNV
HRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQAqi9LycxcXKNSDXaPkCKvw7RQy
iMBDGm1kIY++p3tzbUGuaeu85TsswKnqd50AullEU+aQxRRJGfR8eSKzQJMBXLMQ
b4ptYCc5OrZtRHT8NaZ/df2tc6I88kN8dBu6ybcNGsevXA/iNX3kKLW7naxdr5jj
KUudWSuqDCjCmQa5bYb9H6DreLH2lUItSWBa/YmeZ3VSezDCd+XYO53QKwZVj8Jb
bulZmoo7e7HO1qecEzWKL10UYyEbG3UDPtw+NZc142ZYeEhXQ0dsstGAO5hf3hEl
kQyKGUTpDbKLuyYMFsoH73YLjBqNe+UEhPwE+FWpcky1Sp9RTx/oMLpiZaPR
-----END CERTIFICATE-----`,
shouldFail: true,
},
{
password: "",
privateKey: `-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA4K9Qq7vMY2bGkrdFAYpBYNLlCgnnFU+0pi+N+3bjuWmfX/kw
WXBa3SDqKD08PWWzwvBSLPCCUV2IuUd7tBa1pJ2wXkdoDeI5InYHJKrXbSZonni6
Bex7sgnqV/9o8xFkSOleoQWZgyeKGxtt0J/Z+zhpH+zXahwM4wOL3yzLSQt+NCKM
6N96zXYi16DEa89fYwRxPwE1XTRc7Ddggqx+4iRHvYG0fyTNcPB/+UiFw59EE1Sg
QIyTVntVqpsb6s8XdkFxURoLxefhcMVf2kU0T04OWI3gmeavKfTcj8Z2/bjPSsqP
mgkADv9Ru6VnSK/96TW/NwxWJ32PBz6Sbl9LdwIDAQABAoIBABVh+d5uH/RxyoIZ
+PI9kx1A1NVQvfI0RK/wJKYC2YdCuw0qLOTGIY+b20z7DumU7TenIVrvhKdzrFhd
qjMoWh8RdsByMT/pAKD79JATxi64EgrK2IFJ0TfPY8L+JqHDTPT3aK8QVly5/ZW4
1YmePOOAqdiE9Lc/diaApuYVYD9SL/X7fYs1ezOB4oGXoz0rthX77zHMxcEurpK3
VgSnaq7FYTVY7GrFB+ASiAlDIyLwztz08Ijn8aG0QAZ8GFuPGSmPMXWjLwFhRZsa
Gfy5BYiA0bVSnQSPHzAnHu9HyGlsdouVPPvJB3SrvMl+BFhZiUuR8OGSob7z7hfI
hMyHbNECgYEA/gyG7sHAb5mPkhq9JkTv+LrMY5NDZKYcSlbvBlM3kd6Ib3Hxl+6T
FMq2TWIrh2+mT1C14htziHd05dF6St995Tby6CJxTj6a/2Odnfm+JcOou/ula4Sz
92nIGlGPTJXstDbHGnRCpk6AomXK02stydTyrCisOw1H+LyTG6aT0q8CgYEA4mkO
hfLJkgmJzWIhxHR901uWHz/LId0gC6FQCeaqWmRup6Bl97f0U6xokw4tw8DJOncF
yZpYRXUXhdv/FXCjtXvAhKIX5+e+3dlzPHIdekSfcY00ip/ifAS1OyVviJia+cna
eJgq8WLHxJZim9Ah93NlPyiqGPwtasub90qjZbkCgYEA35WK02o1wII3dvCNc7bM
M+3CoAglEdmXoF1uM/TdPUXKcbqoU3ymeXAGjYhOov3CMp/n0z0xqvLnMLPxmx+i
ny6DDYXyjlhO9WFogHYhwP636+mHJl8+PAsfDvqk0VRJZDmpdUDIv7DrSQGpRfRX
8f+2K4oIOlhv9RuRpI4wHwUCgYB8OjaMyn1NEsy4k2qBt4U+jhcdyEv1pbWqi/U1
qYm5FTgd44VvWVDHBGdQoMv9h28iFCJpzrU2Txv8B4y7v9Ujg+ZLIAFL7j0szt5K
wTZpWvO9Q0Qb98Q2VgL2lADRiyIlglrMJnoRfiisNfOfGKE6e+eGsxI5qUxmN5e5
JQvoiQKBgQCqgyuUBIu/Qsb3qUED/o0S5wCel43Yh/Rl+mxDinOUvJfKJSW2SyEk
+jDo0xw3Opg6ZC5Lj2V809LA/XteaIuyhRuqOopjhHIvIvrYGe+2O8q9/Mv40BYW
0BhJ/Gdseps0C6Z5mTT5Fee4YVlGZuyuNKmKTd4JmqInfBV3ncMWQg==
-----END RSA PRIVATE KEY-----`,
certificate: `-----BEGIN CERTIFICATE-----
MIIDiTCCAnGgAwIBAgIJAIb84Z5Mh31iMA0GCSqGSIb3DQEBCwUAMFsxCzAJBgNV
BAYTAlVTMQ4wDAYDVQQIDAVzdGF0ZTERMA8GA1UEBwwIbG9jYXRpb24xFTATBgNV
BAoMDG9yZ2FuaXphdGlvbjESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTE3MTIxODE4
NTcyM1oXDTI3MTIxNjE4NTcyM1owWzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBXN0
YXRlMREwDwYDVQQHDAhsb2NhdGlvbjEVMBMGA1UECgwMb3JnYW5pemF0aW9uMRIw
EAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQDgr1Cru8xjZsaSt0UBikFg0uUKCecVT7SmL437duO5aZ9f+TBZcFrdIOooPTw9
ZbPC8FIs8IJRXYi5R3u0FrWknbBeR2gN4jkidgckqtdtJmieeLoF7HuyCepX/2jz
EWRI6V6hBZmDJ4obG23Qn9n7OGkf7NdqHAzjA4vfLMtJC340Iozo33rNdiLXoMRr
z19jBHE/ATVdNFzsN2CCrH7iJEe9gbR/JM1w8H/5SIXDn0QTVKBAjJNWe1Wqmxvq
zxd2QXFRGgvF5+FwxV/aRTRPTg5YjeCZ5q8p9NyPxnb9uM9Kyo+aCQAO/1G7pWdI
r/3pNb83DFYnfY8HPpJuX0t3AgMBAAGjUDBOMB0GA1UdDgQWBBQ2/bSCHscnoV+0
d+YJxLu4XLSNIDAfBgNVHSMEGDAWgBQ2/bSCHscnoV+0d+YJxLu4XLSNIDAMBgNV
HRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQC6p4gPwmkoDtRsP1c8IWgXFka+
Q59oe79ZK1RqDE6ZZu0rgw07rPzKr4ofW4hTxnx7PUgKOhWLq9VvwEC/9tDbD0Gw
SKknRZZOiEE3qUZbwNtHMd4UBzpzChTRC6RcwC5zT1/WICMUHxa4b8E2umJuf3Qd
5Y23sXEESx5evr49z6DLcVe2i70o2wJeWs2kaXqhCJt0X7z0rnYqjfFdvxd8dyzt
1DXmE45cLadpWHDg26DMsdchamgnqEo79YUxkH6G/Cb8ZX4igQ/CsxCDOKvccjHO
OncDtuIpK8O7OyfHP3+MBpUFG4P6Ctn7RVcZe9fQweTpfAy18G+loVzuUeOD
-----END CERTIFICATE-----`,
shouldFail: false,
},
{
password: "foobar",
privateKey: `-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA4K9Qq7vMY2bGkrdFAYpBYNLlCgnnFU+0pi+N+3bjuWmfX/kw
WXBa3SDqKD08PWWzwvBSLPCCUV2IuUd7tBa1pJ2wXkdoDeI5InYHJKrXbSZonni6
Bex7sgnqV/9o8xFkSOleoQWZgyeKGxtt0J/Z+zhpH+zXahwM4wOL3yzLSQt+NCKM
6N96zXYi16DEa89fYwRxPwE1XTRc7Ddggqx+4iRHvYG0fyTNcPB/+UiFw59EE1Sg
QIyTVntVqpsb6s8XdkFxURoLxefhcMVf2kU0T04OWI3gmeavKfTcj8Z2/bjPSsqP
mgkADv9Ru6VnSK/96TW/NwxWJ32PBz6Sbl9LdwIDAQABAoIBABVh+d5uH/RxyoIZ
+PI9kx1A1NVQvfI0RK/wJKYC2YdCuw0qLOTGIY+b20z7DumU7TenIVrvhKdzrFhd
qjMoWh8RdsByMT/pAKD79JATxi64EgrK2IFJ0TfPY8L+JqHDTPT3aK8QVly5/ZW4
1YmePOOAqdiE9Lc/diaApuYVYD9SL/X7fYs1ezOB4oGXoz0rthX77zHMxcEurpK3
VgSnaq7FYTVY7GrFB+ASiAlDIyLwztz08Ijn8aG0QAZ8GFuPGSmPMXWjLwFhRZsa
Gfy5BYiA0bVSnQSPHzAnHu9HyGlsdouVPPvJB3SrvMl+BFhZiUuR8OGSob7z7hfI
hMyHbNECgYEA/gyG7sHAb5mPkhq9JkTv+LrMY5NDZKYcSlbvBlM3kd6Ib3Hxl+6T
FMq2TWIrh2+mT1C14htziHd05dF6St995Tby6CJxTj6a/2Odnfm+JcOou/ula4Sz
92nIGlGPTJXstDbHGnRCpk6AomXK02stydTyrCisOw1H+LyTG6aT0q8CgYEA4mkO
hfLJkgmJzWIhxHR901uWHz/LId0gC6FQCeaqWmRup6Bl97f0U6xokw4tw8DJOncF
yZpYRXUXhdv/FXCjtXvAhKIX5+e+3dlzPHIdekSfcY00ip/ifAS1OyVviJia+cna
eJgq8WLHxJZim9Ah93NlPyiqGPwtasub90qjZbkCgYEA35WK02o1wII3dvCNc7bM
M+3CoAglEdmXoF1uM/TdPUXKcbqoU3ymeXAGjYhOov3CMp/n0z0xqvLnMLPxmx+i
ny6DDYXyjlhO9WFogHYhwP636+mHJl8+PAsfDvqk0VRJZDmpdUDIv7DrSQGpRfRX
8f+2K4oIOlhv9RuRpI4wHwUCgYB8OjaMyn1NEsy4k2qBt4U+jhcdyEv1pbWqi/U1
qYm5FTgd44VvWVDHBGdQoMv9h28iFCJpzrU2Txv8B4y7v9Ujg+ZLIAFL7j0szt5K
wTZpWvO9Q0Qb98Q2VgL2lADRiyIlglrMJnoRfiisNfOfGKE6e+eGsxI5qUxmN5e5
JQvoiQKBgQCqgyuUBIu/Qsb3qUED/o0S5wCel43Yh/Rl+mxDinOUvJfKJSW2SyEk
+jDo0xw3Opg6ZC5Lj2V809LA/XteaIuyhRuqOopjhHIvIvrYGe+2O8q9/Mv40BYW
0BhJ/Gdseps0C6Z5mTT5Fee4YVlGZuyuNKmKTd4JmqInfBV3ncMWQg==
-----END RSA PRIVATE KEY-----`,
certificate: `-----BEGIN CERTIFICATE-----
MIIDiTCCAnGgAwIBAgIJAIb84Z5Mh31iMA0GCSqGSIb3DQEBCwUAMFsxCzAJBgNV
BAYTAlVTMQ4wDAYDVQQIDAVzdGF0ZTERMA8GA1UEBwwIbG9jYXRpb24xFTATBgNV
BAoMDG9yZ2FuaXphdGlvbjESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTE3MTIxODE4
NTcyM1oXDTI3MTIxNjE4NTcyM1owWzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBXN0
YXRlMREwDwYDVQQHDAhsb2NhdGlvbjEVMBMGA1UECgwMb3JnYW5pemF0aW9uMRIw
EAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQDgr1Cru8xjZsaSt0UBikFg0uUKCecVT7SmL437duO5aZ9f+TBZcFrdIOooPTw9
ZbPC8FIs8IJRXYi5R3u0FrWknbBeR2gN4jkidgckqtdtJmieeLoF7HuyCepX/2jz
EWRI6V6hBZmDJ4obG23Qn9n7OGkf7NdqHAzjA4vfLMtJC340Iozo33rNdiLXoMRr
z19jBHE/ATVdNFzsN2CCrH7iJEe9gbR/JM1w8H/5SIXDn0QTVKBAjJNWe1Wqmxvq
zxd2QXFRGgvF5+FwxV/aRTRPTg5YjeCZ5q8p9NyPxnb9uM9Kyo+aCQAO/1G7pWdI
r/3pNb83DFYnfY8HPpJuX0t3AgMBAAGjUDBOMB0GA1UdDgQWBBQ2/bSCHscnoV+0
d+YJxLu4XLSNIDAfBgNVHSMEGDAWgBQ2/bSCHscnoV+0d+YJxLu4XLSNIDAMBgNV
HRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQC6p4gPwmkoDtRsP1c8IWgXFka+
Q59oe79ZK1RqDE6ZZu0rgw07rPzKr4ofW4hTxnx7PUgKOhWLq9VvwEC/9tDbD0Gw
SKknRZZOiEE3qUZbwNtHMd4UBzpzChTRC6RcwC5zT1/WICMUHxa4b8E2umJuf3Qd
5Y23sXEESx5evr49z6DLcVe2i70o2wJeWs2kaXqhCJt0X7z0rnYqjfFdvxd8dyzt
1DXmE45cLadpWHDg26DMsdchamgnqEo79YUxkH6G/Cb8ZX4igQ/CsxCDOKvccjHO
OncDtuIpK8O7OyfHP3+MBpUFG4P6Ctn7RVcZe9fQweTpfAy18G+loVzuUeOD
-----END CERTIFICATE-----`,
shouldFail: false,
},
}
+8 -6
View File
@@ -76,9 +76,6 @@ func handleCommonEnvVars() {
globalProfiler = startProfiler(profiler)
}
// Check if object cache is disabled.
globalXLObjCacheDisabled = strings.EqualFold(os.Getenv("_MINIO_CACHE"), "off")
accessKey := os.Getenv("MINIO_ACCESS_KEY")
secretKey := os.Getenv("MINIO_SECRET_KEY")
if accessKey != "" && secretKey != "" {
@@ -102,7 +99,12 @@ func handleCommonEnvVars() {
globalIsBrowserEnabled = bool(browserFlag)
}
globalHTTPTrace = os.Getenv("MINIO_HTTP_TRACE") != ""
traceFile := os.Getenv("MINIO_HTTP_TRACE")
if traceFile != "" {
var err error
globalHTTPTraceFile, err = os.OpenFile(traceFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0660)
fatalIf(err, "error opening file %s", traceFile)
}
globalDomainName = os.Getenv("MINIO_DOMAIN")
if globalDomainName != "" {
@@ -132,13 +134,13 @@ func handleCommonEnvVars() {
// Validation is done after parsing both the storage classes. This is needed because we need one
// storage class value to deduce the correct value of the other storage class.
if globalRRStorageClass.Scheme != "" {
err := validateRRSParity(globalRRStorageClass.Parity, globalStandardStorageClass.Parity)
err = validateParity(globalStandardStorageClass.Parity, globalRRStorageClass.Parity)
fatalIf(err, "Invalid value set in environment variable %s.", reducedRedundancyStorageClassEnv)
globalIsStorageClass = true
}
if globalStandardStorageClass.Scheme != "" {
err := validateSSParity(globalStandardStorageClass.Parity, globalRRStorageClass.Parity)
err = validateParity(globalStandardStorageClass.Parity, globalRRStorageClass.Parity)
fatalIf(err, "Invalid value set in environment variable %s.", standardStorageClassEnv)
globalIsStorageClass = true
}
+56 -67
View File
@@ -20,7 +20,7 @@ import (
"errors"
"fmt"
"io/ioutil"
"strconv"
"reflect"
"sync"
"github.com/minio/minio/pkg/auth"
@@ -49,34 +49,22 @@ var (
// GetVersion get current config version.
func (s *serverConfig) GetVersion() string {
s.RLock()
defer s.RUnlock()
return s.Version
}
// SetRegion set a new region.
func (s *serverConfig) SetRegion(region string) {
s.Lock()
defer s.Unlock()
// Save new region.
s.Region = region
}
// GetRegion get current region.
func (s *serverConfig) GetRegion() string {
s.RLock()
defer s.RUnlock()
return s.Region
}
// SetCredential sets new credential and returns the previous credential.
func (s *serverConfig) SetCredential(creds auth.Credentials) (prevCred auth.Credentials) {
s.Lock()
defer s.Unlock()
// Save previous credential.
prevCred = s.Credential
@@ -89,91 +77,92 @@ func (s *serverConfig) SetCredential(creds auth.Credentials) (prevCred auth.Cred
// GetCredentials get current credentials.
func (s *serverConfig) GetCredential() auth.Credentials {
s.RLock()
defer s.RUnlock()
return s.Credential
}
// SetBrowser set if browser is enabled.
func (s *serverConfig) SetBrowser(b bool) {
s.Lock()
defer s.Unlock()
// Set the new value.
s.Browser = BrowserFlag(b)
}
func (s *serverConfig) SetStorageClass(standardClass, rrsClass storageClass) {
s.Lock()
defer s.Unlock()
// Set the values
s.StorageClass.Standard = standardClass.Scheme + strconv.Itoa(standardClass.Parity)
s.StorageClass.RRS = rrsClass.Scheme + strconv.Itoa(rrsClass.Parity)
s.StorageClass.Standard = standardClass
s.StorageClass.RRS = rrsClass
}
func (s *serverConfig) GetStorageClass() (standardStorageClass, rrsStorageClass storageClass) {
s.RLock()
defer s.RUnlock()
var err error
var ssc storageClass
var rrsc storageClass
if s.StorageClass.Standard != "" {
// Parse the values read from config file into storageClass struct
ssc, err = parseStorageClass(s.StorageClass.Standard)
fatalIf(err, "Invalid value %s set in config.json", s.StorageClass.Standard)
}
if s.StorageClass.RRS != "" {
// Parse the values read from config file into storageClass struct
rrsc, err = parseStorageClass(s.StorageClass.RRS)
fatalIf(err, "Invalid value %s set in config.json", s.StorageClass.RRS)
}
// Validation is done after parsing both the storage classes. This is needed because we need one
// storage class value to deduce the correct value of the other storage class.
if rrsc.Scheme != "" {
err := validateRRSParity(rrsc.Parity, ssc.Parity)
fatalIf(err, "Invalid value %s set in config.json", s.StorageClass.RRS)
globalIsStorageClass = true
}
if ssc.Scheme != "" {
err := validateSSParity(ssc.Parity, rrsc.Parity)
fatalIf(err, "Invalid value %s set in config.json", s.StorageClass.Standard)
globalIsStorageClass = true
}
return
// GetStorageClass reads storage class fields from current config.
// It returns the standard and reduced redundancy storage class struct
func (s *serverConfig) GetStorageClass() (storageClass, storageClass) {
return s.StorageClass.Standard, s.StorageClass.RRS
}
// GetCredentials get current credentials.
func (s *serverConfig) GetBrowser() bool {
s.RLock()
defer s.RUnlock()
return bool(s.Browser)
}
// Save config.
func (s *serverConfig) Save() error {
s.RLock()
defer s.RUnlock()
// Save config file.
return quick.Save(getConfigFile(), s)
}
// Returns the string describing a difference with the given
// configuration object. If the given configuration object is
// identical, an empty string is returned.
func (s *serverConfig) ConfigDiff(t *serverConfig) string {
switch {
case t == nil:
return "Given configuration is empty"
case s.Credential != t.Credential:
return "Credential configuration differs"
case s.Region != t.Region:
return "Region configuration differs"
case s.Browser != t.Browser:
return "Browser configuration differs"
case s.Domain != t.Domain:
return "Domain configuration differs"
case s.StorageClass != t.StorageClass:
return "StorageClass configuration differs"
case !reflect.DeepEqual(s.Notify.AMQP, t.Notify.AMQP):
return "AMQP Notification configuration differs"
case !reflect.DeepEqual(s.Notify.NATS, t.Notify.NATS):
return "NATS Notification configuration differs"
case !reflect.DeepEqual(s.Notify.ElasticSearch, t.Notify.ElasticSearch):
return "ElasticSearch Notification configuration differs"
case !reflect.DeepEqual(s.Notify.Redis, t.Notify.Redis):
return "Redis Notification configuration differs"
case !reflect.DeepEqual(s.Notify.PostgreSQL, t.Notify.PostgreSQL):
return "PostgreSQL Notification configuration differs"
case !reflect.DeepEqual(s.Notify.Kafka, t.Notify.Kafka):
return "Kafka Notification configuration differs"
case !reflect.DeepEqual(s.Notify.Webhook, t.Notify.Webhook):
return "Webhook Notification configuration differs"
case !reflect.DeepEqual(s.Notify.MySQL, t.Notify.MySQL):
return "MySQL Notification configuration differs"
case !reflect.DeepEqual(s.Notify.MQTT, t.Notify.MQTT):
return "MQTT Notification configuration differs"
case reflect.DeepEqual(s, t):
return ""
default:
// This case will not happen unless this comparison
// function has become stale.
return "Configuration differs"
}
}
func newServerConfig() *serverConfig {
srvCfg := &serverConfig{
Version: serverConfigVersion,
Credential: auth.MustGetNewCredentials(),
Region: globalMinioDefaultRegion,
Browser: true,
Notify: &notifier{},
StorageClass: storageClassConfig{
Standard: storageClass{},
RRS: storageClass{},
},
Notify: notifier{},
}
// Make sure to initialize notification configs.
+90
View File
@@ -23,6 +23,7 @@ import (
"reflect"
"testing"
"github.com/minio/minio/pkg/auth"
"github.com/tidwall/gjson"
)
@@ -316,3 +317,92 @@ func TestValidateConfig(t *testing.T) {
}
}
func TestConfigDiff(t *testing.T) {
testCases := []struct {
s, t *serverConfig
diff string
}{
// 1
{&serverConfig{}, nil, "Given configuration is empty"},
// 2
{
&serverConfig{Credential: auth.Credentials{"u1", "p1"}},
&serverConfig{Credential: auth.Credentials{"u1", "p2"}},
"Credential configuration differs",
},
// 3
{&serverConfig{Region: "us-east-1"}, &serverConfig{Region: "us-west-1"}, "Region configuration differs"},
// 4
{&serverConfig{Browser: false}, &serverConfig{Browser: true}, "Browser configuration differs"},
// 5
{&serverConfig{Domain: "domain1"}, &serverConfig{Domain: "domain2"}, "Domain configuration differs"},
// 6
{
&serverConfig{StorageClass: storageClassConfig{storageClass{"1", 8}, storageClass{"2", 6}}},
&serverConfig{StorageClass: storageClassConfig{storageClass{"1", 8}, storageClass{"2", 4}}},
"StorageClass configuration differs",
},
// 7
{
&serverConfig{Notify: notifier{AMQP: map[string]amqpNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{AMQP: map[string]amqpNotify{"1": {Enable: false}}}},
"AMQP Notification configuration differs",
},
// 8
{
&serverConfig{Notify: notifier{NATS: map[string]natsNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{NATS: map[string]natsNotify{"1": {Enable: false}}}},
"NATS Notification configuration differs",
},
// 9
{
&serverConfig{Notify: notifier{ElasticSearch: map[string]elasticSearchNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{ElasticSearch: map[string]elasticSearchNotify{"1": {Enable: false}}}},
"ElasticSearch Notification configuration differs",
},
// 10
{
&serverConfig{Notify: notifier{Redis: map[string]redisNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{Redis: map[string]redisNotify{"1": {Enable: false}}}},
"Redis Notification configuration differs",
},
// 11
{
&serverConfig{Notify: notifier{PostgreSQL: map[string]postgreSQLNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{PostgreSQL: map[string]postgreSQLNotify{"1": {Enable: false}}}},
"PostgreSQL Notification configuration differs",
},
// 12
{
&serverConfig{Notify: notifier{Kafka: map[string]kafkaNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{Kafka: map[string]kafkaNotify{"1": {Enable: false}}}},
"Kafka Notification configuration differs",
},
// 13
{
&serverConfig{Notify: notifier{Webhook: map[string]webhookNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{Webhook: map[string]webhookNotify{"1": {Enable: false}}}},
"Webhook Notification configuration differs",
},
// 14
{
&serverConfig{Notify: notifier{MySQL: map[string]mySQLNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{MySQL: map[string]mySQLNotify{"1": {Enable: false}}}},
"MySQL Notification configuration differs",
},
// 15
{
&serverConfig{Notify: notifier{MQTT: map[string]mqttNotify{"1": {Enable: true}}}},
&serverConfig{Notify: notifier{MQTT: map[string]mqttNotify{"1": {Enable: false}}}},
"MQTT Notification configuration differs",
},
}
for i, testCase := range testCases {
got := testCase.s.ConfigDiff(testCase.t)
if got != testCase.diff {
t.Errorf("Test %d: got %s expected %s", i+1, got, testCase.diff)
}
}
}
+1 -9
View File
@@ -1147,14 +1147,6 @@ func migrateV15ToV16() error {
// Load browser config from existing config in the file.
srvConfig.Browser = cv15.Browser
// Migrate console and file fields
if cv15.Logger.Console.Enable {
srvConfig.Logger.Console = NewConsoleLogger()
}
if cv15.Logger.File.Enable {
srvConfig.Logger.File = NewFileLogger(cv15.Logger.File.Filename)
}
if err = quick.Save(configFile, srvConfig); err != nil {
return fmt.Errorf("Failed to migrate config from ‘%s’ to ‘%s’. %v", cv15.Version, srvConfig.Version, err)
}
@@ -1725,7 +1717,7 @@ func migrateV21ToV22() error {
// Copy over fields from V21 into V22 config struct
srvConfig := &serverConfigV22{
Notify: &notifier{},
Notify: notifier{},
}
srvConfig.Version = serverConfigVersion
srvConfig.Credential = cv21.Credential
+20 -3
View File
@@ -418,6 +418,21 @@ type serverConfigV15 struct {
Notify *notifier `json:"notify"`
}
// FileLogger is introduced to workaround the dependency about logrus
type FileLogger struct {
Enable bool `json:"enable"`
Filename string `json:"filename"`
}
// ConsoleLogger is introduced to workaround the dependency about logrus
type ConsoleLogger struct {
Enable bool `json:"enable"`
}
// Loggers struct is defined with FileLogger and ConsoleLogger
// although they are removed from logging logic. They are
// kept here just to workaround the dependency migration
// code/logic has on them.
type loggers struct {
sync.RWMutex
Console ConsoleLogger `json:"console"`
@@ -532,9 +547,11 @@ type serverConfigV21 struct {
}
// serverConfigV22 is just like version '21' with added support
// for StorageClass
// for StorageClass.
//
// IMPORTANT NOTE: When updating this struct make sure that
// serverConfig.ConfigDiff() is updated as necessary.
type serverConfigV22 struct {
sync.RWMutex
Version string `json:"version"`
// S3 API configuration.
@@ -547,5 +564,5 @@ type serverConfigV22 struct {
StorageClass storageClassConfig `json:"storageclass"`
// Notification queue configuration.
Notify *notifier `json:"notify"`
Notify notifier `json:"notify"`
}
-74
View File
@@ -1,74 +0,0 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"fmt"
"os"
"github.com/Sirupsen/logrus"
)
// ConsoleLogger - console logger which logs into stderr.
type ConsoleLogger struct {
BaseLogTarget
}
// Fire - log entry handler.
func (logger ConsoleLogger) Fire(entry *logrus.Entry) error {
if !logger.Enable {
return nil
}
msgBytes, err := logger.formatter.Format(entry)
if err == nil {
fmt.Fprintf(os.Stderr, string(msgBytes))
}
return err
}
// String - represents ConsoleLogger as string.
func (logger ConsoleLogger) String() string {
enableStr := "disabled"
if logger.Enable {
enableStr = "enabled"
}
return fmt.Sprintf("console:%s", enableStr)
}
// NewConsoleLogger - return new console logger object.
func NewConsoleLogger() (logger ConsoleLogger) {
logger.Enable = true
logger.formatter = new(logrus.TextFormatter)
return logger
}
// InitConsoleLogger - initializes console logger.
func InitConsoleLogger(logger *ConsoleLogger) {
if !logger.Enable {
return
}
if logger.formatter == nil {
logger.formatter = new(logrus.TextFormatter)
}
return
}
+9
View File
@@ -186,6 +186,15 @@ func (endpoints EndpointList) IsHTTPS() bool {
return endpoints[0].IsHTTPS()
}
// GetString - returns endpoint string of i-th endpoint (0-based),
// and empty string for invalid indexes.
func (endpoints EndpointList) GetString(i int) string {
if i < 0 || i >= len(endpoints) {
return ""
}
return endpoints[i].String()
}
// NewEndpointList - returns new endpoint list based on input args.
func NewEndpointList(args ...string) (endpoints EndpointList, err error) {
// isValidDistribution - checks whether given count is a valid distribution for erasure coding.
+2 -2
View File
@@ -70,7 +70,7 @@ func TestErasureCreateFile(t *testing.T) {
if err != nil {
t.Fatalf("Test %d: failed to create test setup: %v", i, err)
}
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks)
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks, test.blocksize)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create ErasureStorage: %v", i, err)
@@ -125,7 +125,7 @@ func benchmarkErasureWrite(data, parity, dataDown, parityDown int, size int64, b
b.Fatalf("failed to create test setup: %v", err)
}
defer setup.Remove()
storage, err := NewErasureStorage(setup.disks, data, parity)
storage, err := NewErasureStorage(setup.disks, data, parity, blockSizeV1)
if err != nil {
b.Fatalf("failed to create ErasureStorage: %v", err)
}
+1 -1
View File
@@ -74,7 +74,7 @@ func TestErasureHealFile(t *testing.T) {
if err != nil {
t.Fatalf("Test %d: failed to setup XL environment: %v", i, err)
}
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.disks-test.dataBlocks)
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.disks-test.dataBlocks, test.blocksize)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create ErasureStorage: %v", i, err)
+3 -3
View File
@@ -86,7 +86,7 @@ func TestErasureReadFile(t *testing.T) {
if err != nil {
t.Fatalf("Test %d: failed to create test setup: %v", i, err)
}
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks)
storage, err := NewErasureStorage(setup.disks, test.dataBlocks, test.onDisks-test.dataBlocks, test.blocksize)
if err != nil {
setup.Remove()
t.Fatalf("Test %d: failed to create ErasureStorage: %v", i, err)
@@ -174,7 +174,7 @@ func TestErasureReadFileRandomOffsetLength(t *testing.T) {
}
defer setup.Remove()
storage, err := NewErasureStorage(setup.disks, dataBlocks, parityBlocks)
storage, err := NewErasureStorage(setup.disks, dataBlocks, parityBlocks, blockSize)
if err != nil {
t.Fatalf("failed to create ErasureStorage: %v", err)
}
@@ -231,7 +231,7 @@ func benchmarkErasureRead(data, parity, dataDown, parityDown int, size int64, b
b.Fatalf("failed to create test setup: %v", err)
}
defer setup.Remove()
storage, err := NewErasureStorage(setup.disks, data, parity)
storage, err := NewErasureStorage(setup.disks, data, parity, blockSizeV1)
if err != nil {
b.Fatalf("failed to create ErasureStorage: %v", err)
}
+3 -2
View File
@@ -44,8 +44,9 @@ type ErasureStorage struct {
// NewErasureStorage creates a new ErasureStorage. The storage erasure codes and protects all data written to
// the disks.
func NewErasureStorage(disks []StorageAPI, dataBlocks, parityBlocks int) (s ErasureStorage, err error) {
erasure, err := reedsolomon.New(dataBlocks, parityBlocks)
func NewErasureStorage(disks []StorageAPI, dataBlocks, parityBlocks int, blockSize int64) (s ErasureStorage, err error) {
shardsize := (int(blockSize) + dataBlocks - 1) / dataBlocks
erasure, err := reedsolomon.New(dataBlocks, parityBlocks, reedsolomon.WithAutoGoroutines(shardsize))
if err != nil {
return s, errors.Tracef("failed to create erasure coding: %v", err)
}
+1 -1
View File
@@ -52,7 +52,7 @@ func TestErasureDecode(t *testing.T) {
copy(buffer, data)
disks := make([]StorageAPI, test.dataBlocks+test.parityBlocks)
storage, err := NewErasureStorage(disks, test.dataBlocks, test.parityBlocks)
storage, err := NewErasureStorage(disks, test.dataBlocks, test.parityBlocks, blockSizeV1)
if err != nil {
t.Fatalf("Test %d: failed to create erasure storage: %v", i, err)
}
+2 -40
View File
@@ -365,15 +365,8 @@ func loadNotificationConfig(bucket string, objAPI ObjectLayer) (*notificationCon
// Construct the notification config path.
ncPath := path.Join(bucketConfigPrefix, bucket, bucketNotificationConfig)
// Acquire a write lock on notification config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, ncPath)
if err := objLock.GetRLock(globalOperationTimeout); err != nil {
return nil, err
}
defer objLock.RUnlock()
var buffer bytes.Buffer
err := objAPI.GetObject(minioMetaBucket, ncPath, 0, -1, &buffer) // Read everything.
err := objAPI.GetObject(minioMetaBucket, ncPath, 0, -1, &buffer, "") // Read everything.
if err != nil {
// 'notification.xml' not found return
// 'errNoSuchNotifications'. This is default when no
@@ -416,15 +409,8 @@ func loadListenerConfig(bucket string, objAPI ObjectLayer) ([]listenerConfig, er
// Construct the notification config path.
lcPath := path.Join(bucketConfigPrefix, bucket, bucketListenerConfig)
// Acquire a write lock on notification config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, lcPath)
if err := objLock.GetRLock(globalOperationTimeout); err != nil {
return nil, err
}
defer objLock.RUnlock()
var buffer bytes.Buffer
err := objAPI.GetObject(minioMetaBucket, lcPath, 0, -1, &buffer)
err := objAPI.GetObject(minioMetaBucket, lcPath, 0, -1, &buffer, "")
if err != nil {
// 'listener.json' not found return
// 'errNoSuchNotifications'. This is default when no
@@ -463,12 +449,6 @@ func persistNotificationConfig(bucket string, ncfg *notificationConfig, obj Obje
// build path
ncPath := path.Join(bucketConfigPrefix, bucket, bucketNotificationConfig)
// Acquire a write lock on notification config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, ncPath)
if err = objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
defer objLock.Unlock()
// write object to path
hashReader, err := hash.NewReader(bytes.NewReader(buf), int64(len(buf)), "", getSHA256Hash(buf))
@@ -494,12 +474,6 @@ func persistListenerConfig(bucket string, lcfg []listenerConfig, obj ObjectLayer
// build path
lcPath := path.Join(bucketConfigPrefix, bucket, bucketListenerConfig)
// Acquire a write lock on notification config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, lcPath)
if err = objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
defer objLock.Unlock()
// write object to path
hashReader, err := hash.NewReader(bytes.NewReader(buf), int64(len(buf)), "", getSHA256Hash(buf))
@@ -526,12 +500,6 @@ func removeNotificationConfig(bucket string, objAPI ObjectLayer) error {
ncPath := path.Join(bucketConfigPrefix, bucket, bucketNotificationConfig)
// Acquire a write lock on notification config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, ncPath)
if err := objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
defer objLock.Unlock()
return objAPI.DeleteObject(minioMetaBucket, ncPath)
}
@@ -540,12 +508,6 @@ func removeListenerConfig(bucket string, objAPI ObjectLayer) error {
// make the path
lcPath := path.Join(bucketConfigPrefix, bucket, bucketListenerConfig)
// Acquire a write lock on notification config before modifying.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, lcPath)
if err := objLock.GetLock(globalOperationTimeout); err != nil {
return err
}
defer objLock.Unlock()
return objAPI.DeleteObject(minioMetaBucket, lcPath)
}
-86
View File
@@ -1,86 +0,0 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"fmt"
"os"
"github.com/Sirupsen/logrus"
)
// FileLogger - file logger which logs to a file.
type FileLogger struct {
BaseLogTarget
Filename string `json:"filename"`
file *os.File
}
// Fire - log entry handler.
func (logger FileLogger) Fire(entry *logrus.Entry) (err error) {
if !logger.Enable {
return nil
}
msgBytes, err := logger.formatter.Format(entry)
if err != nil {
return err
}
if _, err = logger.file.Write(msgBytes); err != nil {
return err
}
err = logger.file.Sync()
return err
}
// String - represents ConsoleLogger as string.
func (logger FileLogger) String() string {
enableStr := "disabled"
if logger.Enable {
enableStr = "enabled"
}
return fmt.Sprintf("file:%s:%s", enableStr, logger.Filename)
}
// NewFileLogger - creates new file logger object.
func NewFileLogger(filename string) (logger FileLogger) {
logger.Enable = true
logger.formatter = new(logrus.JSONFormatter)
logger.Filename = filename
return logger
}
// InitFileLogger - initializes file logger.
func InitFileLogger(logger *FileLogger) (err error) {
if !logger.Enable {
return err
}
if logger.formatter == nil {
logger.formatter = new(logrus.JSONFormatter)
}
if logger.file == nil {
logger.file, err = os.OpenFile(logger.Filename, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0664)
}
return err
}
+260
View File
@@ -0,0 +1,260 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"fmt"
"io"
"os"
"path"
"time"
errors2 "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/lock"
)
// FS format version strings.
const (
formatBackendFS = "fs"
formatFSVersionV1 = "1"
formatFSVersionV2 = "2"
)
// formatFSV1 - structure holds format version '1'.
type formatFSV1 struct {
formatMetaV1
FS struct {
Version string `json:"version"`
} `json:"fs"`
}
// formatFSV2 - structure is same as formatFSV1. But the multipart backend
// structure is flat instead of hierarchy now.
// In .minio.sys/multipart we have:
// sha256(bucket/object)/uploadID/[fs.json, 1.etag, 2.etag ....]
type formatFSV2 = formatFSV1
// Used to detect the version of "fs" format.
type formatFSVersionDetect struct {
FS struct {
Version string `json:"version"`
} `json:"fs"`
}
// Returns the latest "fs" format V1
func newFormatFSV1() (format *formatFSV1) {
f := &formatFSV1{}
f.Version = formatMetaVersionV1
f.Format = formatBackendFS
f.FS.Version = formatFSVersionV1
return f
}
// Returns the latest "fs" format V2
func newFormatFSV2() (format *formatFSV2) {
f := &formatFSV2{}
f.Version = formatMetaVersionV1
f.Format = formatBackendFS
f.FS.Version = formatFSVersionV2
return f
}
// Returns the field formatMetaV1.Format i.e the string "fs" which is never likely to change.
// We do not use this function in XL to get the format as the file is not fcntl-locked on XL.
func formatMetaGetFormatBackendFS(r io.ReadSeeker) (string, error) {
format := &formatMetaV1{}
if err := jsonLoad(r, format); err != nil {
return "", err
}
if format.Version == formatMetaVersionV1 {
return format.Format, nil
}
return "", fmt.Errorf(`format.Version expected: %s, got: %s`, formatMetaVersionV1, format.Version)
}
// Returns formatFS.FS.Version
func formatFSGetVersion(r io.ReadSeeker) (string, error) {
format := &formatFSVersionDetect{}
if err := jsonLoad(r, format); err != nil {
return "", err
}
return format.FS.Version, nil
}
// Migrate from V1 to V2. V2 implements new backend format for multipart
// uploads. Delete the previous multipart directory.
func formatFSMigrateV1ToV2(wlk *lock.LockedFile, fsPath string) error {
version, err := formatFSGetVersion(wlk)
if err != nil {
return err
}
if version != formatFSVersionV1 {
return fmt.Errorf(`format.json version expected %s, found %s`, formatFSVersionV1, version)
}
if err = fsRemoveAll(path.Join(fsPath, minioMetaMultipartBucket)); err != nil {
return err
}
if err = os.MkdirAll(path.Join(fsPath, minioMetaMultipartBucket), 0755); err != nil {
return err
}
return jsonSave(wlk.File, newFormatFSV2())
}
// Migrate the "fs" backend.
// Migration should happen when formatFSV1.FS.Version changes. This version
// can change when there is a change to the struct formatFSV1.FS or if there
// is any change in the backend file system tree structure.
func formatFSMigrate(wlk *lock.LockedFile, fsPath string) error {
// Add any migration code here in case we bump format.FS.Version
version, err := formatFSGetVersion(wlk)
if err != nil {
return err
}
switch version {
case formatFSVersionV1:
if err = formatFSMigrateV1ToV2(wlk, fsPath); err != nil {
return err
}
fallthrough
case formatFSVersionV2:
// We are at the latest version.
}
// Make sure that the version is what we expect after the migration.
version, err = formatFSGetVersion(wlk)
if err != nil {
return err
}
if version != formatFSVersionV2 {
return fmt.Errorf(`%s file: expected FS version: %s, found FS version: %s`, formatConfigFile, formatFSVersionV2, version)
}
return nil
}
// Creates a new format.json if unformatted.
func createFormatFS(fsFormatPath string) error {
// Attempt a write lock on formatConfigFile `format.json`
// file stored in minioMetaBucket(.minio.sys) directory.
lk, err := lock.TryLockedOpenFile(fsFormatPath, os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
return errors2.Trace(err)
}
// Close the locked file upon return.
defer lk.Close()
fi, err := lk.Stat()
if err != nil {
return errors2.Trace(err)
}
if fi.Size() != 0 {
// format.json already got created because of another minio process's createFormatFS()
return nil
}
return jsonSave(lk.File, newFormatFSV1())
}
// This function returns a read-locked format.json reference to the caller.
// The file descriptor should be kept open throughout the life
// of the process so that another minio process does not try to
// migrate the backend when we are actively working on the backend.
func initFormatFS(fsPath string) (rlk *lock.RLockedFile, err error) {
fsFormatPath := pathJoin(fsPath, minioMetaBucket, formatConfigFile)
// Any read on format.json should be done with read-lock.
// Any write on format.json should be done with write-lock.
for {
isEmpty := false
rlk, err := lock.RLockedOpenFile(fsFormatPath)
if err == nil {
// format.json can be empty in a rare condition when another
// minio process just created the file but could not hold lock
// and write to it.
var fi os.FileInfo
fi, err = rlk.Stat()
if err != nil {
return nil, errors2.Trace(err)
}
isEmpty = fi.Size() == 0
}
if os.IsNotExist(err) || isEmpty {
if err == nil {
rlk.Close()
}
// Fresh disk - create format.json
err = createFormatFS(fsFormatPath)
if err == lock.ErrAlreadyLocked {
// Lock already present, sleep and attempt again.
// Can happen in a rare situation when a parallel minio process
// holds the lock and creates format.json
time.Sleep(100 * time.Millisecond)
continue
}
if err != nil {
return nil, errors2.Trace(err)
}
// After successfully creating format.json try to hold a read-lock on
// the file.
continue
}
if err != nil {
return nil, errors2.Trace(err)
}
formatBackend, err := formatMetaGetFormatBackendFS(rlk)
if err != nil {
return nil, errors2.Trace(err)
}
if formatBackend != formatBackendFS {
return nil, fmt.Errorf(`%s file: expected format-type: %s, found: %s`, formatConfigFile, formatBackendFS, formatBackend)
}
version, err := formatFSGetVersion(rlk)
if err != nil {
return nil, err
}
if version != formatFSVersionV2 {
// Format needs migration
rlk.Close()
// Hold write lock during migration so that we do not disturb any
// minio processes running in parallel.
wlk, err := lock.TryLockedOpenFile(fsFormatPath, os.O_RDWR, 0)
if err == lock.ErrAlreadyLocked {
// Lock already present, sleep and attempt again.
wlk.Close()
time.Sleep(100 * time.Millisecond)
continue
}
if err != nil {
return nil, err
}
err = formatFSMigrate(wlk, fsPath)
wlk.Close()
if err != nil {
// Migration failed, bail out so that the user can observe what happened.
return nil, err
}
// Successfully migrated, now try to hold a read-lock on format.json
continue
}
return rlk, nil
}
}
+102
View File
@@ -0,0 +1,102 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"os"
"path/filepath"
"testing"
)
// TestFSFormatFS - tests initFormatFS, formatMetaGetFormatBackendFS, formatFSGetVersion.
func TestFSFormatFS(t *testing.T) {
// Prepare for testing
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
defer os.RemoveAll(disk)
fsFormatPath := pathJoin(disk, minioMetaBucket, formatConfigFile)
// Assign a new UUID.
uuid := mustGetUUID()
// Initialize meta volume, if volume already exists ignores it.
if err := initMetaVolumeFS(disk, uuid); err != nil {
t.Fatal(err)
}
rlk, err := initFormatFS(disk)
if err != nil {
t.Fatal(err)
}
rlk.Close()
// Do the basic sanity checks to check if initFormatFS() did its job.
f, err := os.OpenFile(fsFormatPath, os.O_RDWR, 0)
if err != nil {
t.Fatal(err)
}
defer f.Close()
format, err := formatMetaGetFormatBackendFS(f)
if err != nil {
t.Fatal(err)
}
if format != formatBackendFS {
t.Fatalf(`expected: %s, got: %s`, formatBackendFS, format)
}
version, err := formatFSGetVersion(f)
if err != nil {
t.Fatal(err)
}
if version != formatFSVersionV2 {
t.Fatalf(`expected: %s, got: %s`, formatFSVersionV2, version)
}
// Corrupt the format.json file and test the functions.
// formatMetaGetFormatBackendFS, formatFSGetVersion, initFormatFS should return errors.
if err = f.Truncate(0); err != nil {
t.Fatal(err)
}
if _, err = f.WriteString("b"); err != nil {
t.Fatal(err)
}
if _, err = formatMetaGetFormatBackendFS(f); err == nil {
t.Fatal("expected to fail")
}
if _, err = formatFSGetVersion(rlk); err == nil {
t.Fatal("expected to fail")
}
if _, err = initFormatFS(disk); err == nil {
t.Fatal("expected to fail")
}
// With unknown formatMetaV1.Version formatMetaGetFormatBackendFS, initFormatFS should return error.
if err = f.Truncate(0); err != nil {
t.Fatal(err)
}
// Here we set formatMetaV1.Version to "2"
if _, err = f.WriteString(`{"version":"2","format":"fs","fs":{"version":"1"}}`); err != nil {
t.Fatal(err)
}
if _, err = formatMetaGetFormatBackendFS(f); err == nil {
t.Fatal("expected to fail")
}
if _, err = initFormatFS(disk); err == nil {
t.Fatal("expected to fail")
}
}
+52
View File
@@ -0,0 +1,52 @@
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
// Format related consts
const (
// Format config file carries backend format specific details.
formatConfigFile = "format.json"
// Format config tmp file carries backend format.
formatConfigFileTmp = "format.json.tmp"
)
const (
// Version of the formatMetaV1
formatMetaVersionV1 = "1"
)
// format.json currently has the format:
// {
// "version": "1",
// "format": "XXXXX",
// "XXXXX": {
//
// }
// }
// Here "XXXXX" depends on the backend, currently we have "fs" and "xl" implementations.
// formatMetaV1 should be inherited by backend format structs. Please look at format-fs.go
// and format-xl.go for details.
// Ideally we will never have a situation where we will have to change the
// fields of this struct and deal with related migration.
type formatMetaV1 struct {
// Version of the format config.
Version string `json:"version"`
// Format indicates the backend format type, supports two values 'xl' and 'fs'.
Format string `json:"format"`
}
+131 -278
View File
@@ -20,168 +20,30 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"reflect"
"sync"
errors2 "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/lock"
)
// fsFormat - structure holding 'fs' format.
type fsFormat struct {
Version string `json:"version"`
}
// FS format version strings.
const (
// Represents the current backend disk structure
// version under `.minio.sys` and actual data namespace.
// formatConfigV1.fsFormat.Version
fsFormatBackendV1 = "1"
)
// xlFormat - structure holding 'xl' format.
type xlFormat struct {
Version string `json:"version"` // Version of 'xl' format.
Disk string `json:"disk"` // Disk field carries assigned disk uuid.
// JBOD field carries the input disk order generated the first
// time when fresh disks were supplied.
JBOD []string `json:"jbod"`
}
// XL format version strings.
const (
// Represents the current backend disk structure
// version under `.minio.sys` and actual data namespace.
// formatConfigV1.xlFormat.Version
xlFormatBackendV1 = "1"
)
// formatConfigV1 - structure holds format config version '1'.
type formatConfigV1 struct {
Version string `json:"version"` // Version of the format config.
// Format indicates the backend format type, supports two values 'xl' and 'fs'.
Format string `json:"format"`
FS *fsFormat `json:"fs,omitempty"` // FS field holds fs format.
XL *xlFormat `json:"xl,omitempty"` // XL field holds xl format.
}
// Format json file.
const (
// Format config file carries backend format specific details.
formatConfigFile = "format.json"
// Format config tmp file carries backend format.
formatConfigFileTmp = "format.json.tmp"
)
// `format.json` version value.
const (
// formatConfigV1.Version represents the version string
// of the current structure and its fields in `format.json`.
formatFileV1 = "1"
// Future `format.json` structure changes should have
// its own version and should be subsequently listed here.
)
// Constitutes `format.json` backend name.
const (
// Represents FS backend.
formatBackendFS = "fs"
// Represents XL backend.
formatBackendXL = "xl"
// formatXLV1.XL.Version
formatXLVersionV1 = "1"
)
// CheckFS if the format is FS and is valid with right values
// returns appropriate errors otherwise.
func (f *formatConfigV1) CheckFS() error {
// Validate if format config version is v1.
if f.Version != formatFileV1 {
return fmt.Errorf("Unknown format file version '%s'", f.Version)
}
// Validate if we have the expected format.
if f.Format != formatBackendFS {
return fmt.Errorf("FS backend format required. Found '%s'", f.Format)
}
// Check if format is currently supported.
if f.FS.Version != fsFormatBackendV1 {
return fmt.Errorf("Unknown backend FS format version '%s'", f.FS.Version)
}
// Success.
return nil
}
// LoadFormat - loads format config v1, returns `errUnformattedDisk`
// if reading format.json fails with io.EOF.
func (f *formatConfigV1) LoadFormat(lk *lock.LockedFile) error {
_, err := f.ReadFrom(lk)
if errors2.Cause(err) == io.EOF {
// No data on disk `format.json` still empty
// treat it as unformatted disk.
return errors2.Trace(errUnformattedDisk)
}
return err
}
func (f *formatConfigV1) WriteTo(lk *lock.LockedFile) (n int64, err error) {
// Serialize to prepare to write to disk.
var fbytes []byte
fbytes, err = json.Marshal(f)
if err != nil {
return 0, errors2.Trace(err)
}
if err = lk.Truncate(0); err != nil {
return 0, errors2.Trace(err)
}
_, err = lk.Write(fbytes)
if err != nil {
return 0, errors2.Trace(err)
}
return int64(len(fbytes)), nil
}
func (f *formatConfigV1) ReadFrom(lk *lock.LockedFile) (n int64, err error) {
var fbytes []byte
fi, err := lk.Stat()
if err != nil {
return 0, errors2.Trace(err)
}
fbytes, err = ioutil.ReadAll(io.NewSectionReader(lk, 0, fi.Size()))
if err != nil {
return 0, errors2.Trace(err)
}
if len(fbytes) == 0 {
return 0, errors2.Trace(io.EOF)
}
// Decode `format.json`.
if err = json.Unmarshal(fbytes, f); err != nil {
return 0, errors2.Trace(err)
}
return int64(len(fbytes)), nil
}
func newFSFormat() (format *formatConfigV1) {
return newFSFormatV1()
}
// newFSFormatV1 - initializes new formatConfigV1 with FS format info.
func newFSFormatV1() (format *formatConfigV1) {
return &formatConfigV1{
Version: formatFileV1,
Format: formatBackendFS,
FS: &fsFormat{
Version: fsFormatBackendV1,
},
}
// Represents the current backend disk structure
// version under `.minio.sys` and actual data namespace.
// formatXLV1 - structure holds format config version '1'.
type formatXLV1 struct {
formatMetaV1
XL struct {
Version string `json:"version"` // Version of 'xl' format.
Disk string `json:"disk"` // Disk field carries assigned disk uuid.
// JBOD field carries the input disk order generated the first
// time when fresh disks were supplied.
JBOD []string `json:"jbod"`
} `json:"xl"` // XL field holds xl format.
}
/*
@@ -271,7 +133,7 @@ func formatErrsSummary(errs []error) (formatCount, unformattedDiskCount,
}
// loadAllFormats - load all format config from all input disks in parallel.
func loadAllFormats(bootstrapDisks []StorageAPI) ([]*formatConfigV1, []error) {
func loadAllFormats(bootstrapDisks []StorageAPI) ([]*formatXLV1, []error) {
// Initialize sync waitgroup.
var wg = &sync.WaitGroup{}
@@ -279,38 +141,32 @@ func loadAllFormats(bootstrapDisks []StorageAPI) ([]*formatConfigV1, []error) {
var sErrs = make([]error, len(bootstrapDisks))
// Initialize format configs.
var formatConfigs = make([]*formatConfigV1, len(bootstrapDisks))
var formats = make([]*formatXLV1, len(bootstrapDisks))
// Make a volume entry on all underlying storage disks.
// Load format from each disk in parallel
for index, disk := range bootstrapDisks {
if disk == nil {
sErrs[index] = errDiskNotFound
continue
}
wg.Add(1)
// Make a volume inside a go-routine.
// Launch go-routine per disk.
go func(index int, disk StorageAPI) {
defer wg.Done()
formatConfig, lErr := loadFormat(disk)
format, lErr := loadFormat(disk)
if lErr != nil {
sErrs[index] = lErr
return
}
formatConfigs[index] = formatConfig
formats[index] = format
}(index, disk)
}
// Wait for all make vol to finish.
// Wait for all go-routines to finish.
wg.Wait()
for _, err := range sErrs {
if err != nil {
// Return all formats and errors.
return formatConfigs, sErrs
}
}
// Return all formats and nil
return formatConfigs, sErrs
return formats, sErrs
}
// genericFormatCheckXL - validates and returns error.
@@ -318,7 +174,7 @@ func loadAllFormats(bootstrapDisks []StorageAPI) ([]*formatConfigV1, []error) {
// if (any disk is corrupt) return error // phase2
// if (jbod inconsistent) return error // phase2
// if (disks not recognized) // Always error.
func genericFormatCheckXL(formatConfigs []*formatConfigV1, sErrs []error) (err error) {
func genericFormatCheckXL(formats []*formatXLV1, sErrs []error) (err error) {
// Calculate the errors.
var (
errCorruptFormatCount = 0
@@ -342,20 +198,20 @@ func genericFormatCheckXL(formatConfigs []*formatConfigV1, sErrs []error) (err e
}
// Calculate read quorum.
readQuorum := len(formatConfigs) / 2
readQuorum := len(formats) / 2
// Validate the err count under read quorum.
if errCount > len(formatConfigs)-readQuorum {
if errCount > len(formats)-readQuorum {
return errXLReadQuorum
}
// Check if number of corrupted format under read quorum
if errCorruptFormatCount > len(formatConfigs)-readQuorum {
if errCorruptFormatCount > len(formats)-readQuorum {
return errCorruptedFormat
}
// Validates if format and JBOD are consistent across all disks.
if err = checkFormatXL(formatConfigs); err != nil {
if err = checkFormatXL(formats); err != nil {
return err
}
@@ -366,15 +222,15 @@ func genericFormatCheckXL(formatConfigs []*formatConfigV1, sErrs []error) (err e
// isSavedUUIDInOrder - validates if disk uuid is present and valid in all
// available format config JBOD. This function also validates if the disk UUID
// is always available on all JBOD under the same order.
func isSavedUUIDInOrder(uuid string, formatConfigs []*formatConfigV1) bool {
func isSavedUUIDInOrder(uuid string, formats []*formatXLV1) bool {
var orderIndexes []int
// Validate each for format.json for relevant uuid.
for _, formatConfig := range formatConfigs {
if formatConfig == nil {
for _, format := range formats {
if format == nil {
continue
}
// Validate if UUID is present in JBOD.
uuidIndex := findDiskIndex(uuid, formatConfig.XL.JBOD)
uuidIndex := findDiskIndex(uuid, format.XL.JBOD)
if uuidIndex == -1 {
// UUID not found.
errorIf(errDiskNotFound, "Disk %s not found in JBOD list", uuid)
@@ -398,15 +254,15 @@ func isSavedUUIDInOrder(uuid string, formatConfigs []*formatConfigV1) bool {
}
// checkDisksConsistency - checks if all disks are consistent with all JBOD entries on all disks.
func checkDisksConsistency(formatConfigs []*formatConfigV1) error {
var disks = make([]string, len(formatConfigs))
func checkDisksConsistency(formats []*formatXLV1) error {
var disks = make([]string, len(formats))
// Collect currently available disk uuids.
for index, formatConfig := range formatConfigs {
if formatConfig == nil {
for index, format := range formats {
if format == nil {
disks[index] = ""
continue
}
disks[index] = formatConfig.XL.Disk
disks[index] = format.XL.Disk
}
// Validate collected uuids and verify JBOD.
for _, uuid := range disks {
@@ -414,7 +270,7 @@ func checkDisksConsistency(formatConfigs []*formatConfigV1) error {
continue
}
// Is uuid present on all JBOD ?.
if !isSavedUUIDInOrder(uuid, formatConfigs) {
if !isSavedUUIDInOrder(uuid, formats) {
return fmt.Errorf("%s disk not found in JBOD", uuid)
}
}
@@ -422,17 +278,17 @@ func checkDisksConsistency(formatConfigs []*formatConfigV1) error {
}
// checkJBODConsistency - validate xl jbod order if they are consistent.
func checkJBODConsistency(formatConfigs []*formatConfigV1) error {
func checkJBODConsistency(formats []*formatXLV1) error {
var sentinelJBOD []string
// Extract first valid JBOD.
for _, format := range formatConfigs {
for _, format := range formats {
if format == nil {
continue
}
sentinelJBOD = format.XL.JBOD
break
}
for _, format := range formatConfigs {
for _, format := range formats {
if format == nil {
continue
}
@@ -458,14 +314,14 @@ func findDiskIndex(disk string, jbod []string) int {
// format-config. If assignUUIDs is true, it assigns UUIDs to disks
// with missing format configurations in the reference configuration.
func reorderDisks(bootstrapDisks []StorageAPI,
formatConfigs []*formatConfigV1, assignUUIDs bool) (*formatConfigV1,
formats []*formatXLV1, assignUUIDs bool) (*formatXLV1,
[]StorageAPI, error) {
// Pick first non-nil format-cfg as reference
var refCfg *formatConfigV1
for _, formatConfig := range formatConfigs {
if formatConfig != nil {
refCfg = formatConfig
var refCfg *formatXLV1
for _, format := range formats {
if format != nil {
refCfg = format
break
}
}
@@ -476,7 +332,7 @@ func reorderDisks(bootstrapDisks []StorageAPI,
// construct reordered disk slice
var newDisks = make([]StorageAPI, len(bootstrapDisks))
for fIndex, format := range formatConfigs {
for fIndex, format := range formats {
if format == nil {
continue
}
@@ -501,7 +357,7 @@ func reorderDisks(bootstrapDisks []StorageAPI,
}
// loadFormat - loads format.json from disk.
func loadFormat(disk StorageAPI) (format *formatConfigV1, err error) {
func loadFormat(disk StorageAPI) (format *formatXLV1, err error) {
buf, err := disk.ReadAll(minioMetaBucket, formatConfigFile)
if err != nil {
// 'file not found' and 'volume not found' as
@@ -512,8 +368,10 @@ func loadFormat(disk StorageAPI) (format *formatConfigV1, err error) {
if err != nil {
return nil, err
}
if len(vols) > 1 {
// 'format.json' not found, but we found user data.
if len(vols) > 1 || (len(vols) == 1 &&
vols[0].Name != minioMetaBucket) {
// 'format.json' not found, but we
// found user data.
return nil, errCorruptedFormat
}
// No other data found, its a fresh disk.
@@ -523,7 +381,7 @@ func loadFormat(disk StorageAPI) (format *formatConfigV1, err error) {
}
// Try to decode format json into formatConfigV1 struct.
format = &formatConfigV1{}
format = &formatXLV1{}
if err = json.Unmarshal(buf, format); err != nil {
return nil, err
}
@@ -532,26 +390,22 @@ func loadFormat(disk StorageAPI) (format *formatConfigV1, err error) {
return format, nil
}
// collectNSaveNewFormatConfigs - creates new format configs based on
// the reference config and saves it on all disks, this is to be
// called from healFormatXL* functions.
func collectNSaveNewFormatConfigs(referenceConfig *formatConfigV1,
orderedDisks []StorageAPI) error {
// collectNSaveNewFormatConfigs - generates new format configs based on
// the given ref. config and saves on each disk
func collectNSaveNewFormatConfigs(referenceConfig *formatXLV1,
orderedDisks []StorageAPI, dryRun bool) error {
// Collect new format configs that need to be written.
var newFormatConfigs = make([]*formatConfigV1, len(orderedDisks))
var newFormatConfigs = make([]*formatXLV1, len(orderedDisks))
for index := range orderedDisks {
// New configs are generated since we are going
// to re-populate across all disks.
config := &formatConfigV1{
Version: referenceConfig.Version,
Format: referenceConfig.Format,
XL: &xlFormat{
Version: referenceConfig.XL.Version,
Disk: referenceConfig.XL.JBOD[index],
JBOD: referenceConfig.XL.JBOD,
},
}
config := &formatXLV1{}
config.Version = referenceConfig.Version
config.Format = referenceConfig.Format
config.XL.Version = referenceConfig.XL.Version
config.XL.Disk = referenceConfig.XL.JBOD[index]
config.XL.JBOD = referenceConfig.XL.JBOD
newFormatConfigs[index] = config
}
@@ -561,18 +415,21 @@ func collectNSaveNewFormatConfigs(referenceConfig *formatConfigV1,
}
// Save new `format.json` across all disks, in JBOD order.
return saveFormatXL(orderedDisks, newFormatConfigs)
if !dryRun {
return saveFormatXL(orderedDisks, newFormatConfigs)
}
return nil
}
// Heals any missing format.json on the drives. Returns error only for
// unexpected errors as regular errors can be ignored since there
// might be enough quorum to be operational. Heals only fresh disks.
func healFormatXLFreshDisks(storageDisks []StorageAPI,
formatConfigs []*formatConfigV1) error {
formats []*formatXLV1, dryRun bool) error {
// Reorder the disks based on the JBOD order.
// Reorder disks based on JBOD order, and get reference config.
referenceConfig, orderedDisks, err := reorderDisks(storageDisks,
formatConfigs, true)
formats, true)
if err != nil {
return err
}
@@ -581,22 +438,25 @@ func healFormatXLFreshDisks(storageDisks []StorageAPI,
// We need to make sure we have kept the previous order
// and allowed fresh disks to be arranged anywhere.
// Following block facilitates to put fresh disks.
for index, format := range formatConfigs {
for index, format := range formats {
if format != nil {
continue
}
// Format is missing so we go through ordered disks.
if format == nil {
// At this point when disk is missing the fresh disk
// in the stack get it back from storageDisks.
for oIndex, disk := range orderedDisks {
if disk == nil {
orderedDisks[oIndex] = storageDisks[index]
break
}
// At this point when disk is missing the fresh disk
// in the stack get it back from storageDisks.
for oIndex, disk := range orderedDisks {
if disk == nil {
orderedDisks[oIndex] = storageDisks[index]
break
}
}
}
// apply new format config and save to all disks
return collectNSaveNewFormatConfigs(referenceConfig, orderedDisks)
return collectNSaveNewFormatConfigs(referenceConfig, orderedDisks,
dryRun)
}
// collectUnAssignedDisks - collect disks unassigned to orderedDisks
@@ -624,9 +484,9 @@ func collectUnAssignedDisks(storageDisks, orderedDisks []StorageAPI) (
// Inspect the content of all disks to guess the right order according
// to the format files. The right order is represented in orderedDisks
func reorderDisksByInspection(orderedDisks, storageDisks []StorageAPI,
formatConfigs []*formatConfigV1) ([]StorageAPI, error) {
formats []*formatXLV1) ([]StorageAPI, error) {
for index, format := range formatConfigs {
for index, format := range formats {
if format != nil {
continue
}
@@ -681,11 +541,11 @@ func reorderDisksByInspection(orderedDisks, storageDisks []StorageAPI,
// Heals corrupted format json in all disks
func healFormatXLCorruptedDisks(storageDisks []StorageAPI,
formatConfigs []*formatConfigV1) error {
formats []*formatXLV1, dryRun bool) error {
// Reorder the disks based on the JBOD order.
// Reorder disks based on JBOD order, and update ref. config.
referenceConfig, orderedDisks, err := reorderDisks(storageDisks,
formatConfigs, true)
formats, true)
if err != nil {
return err
}
@@ -693,7 +553,7 @@ func healFormatXLCorruptedDisks(storageDisks []StorageAPI,
// For disks with corrupted formats, inspect the disks
// contents to guess the disks order
orderedDisks, err = reorderDisksByInspection(orderedDisks, storageDisks,
formatConfigs)
formats)
if err != nil {
return err
}
@@ -711,8 +571,9 @@ func healFormatXLCorruptedDisks(storageDisks []StorageAPI,
}
}
// apply new format config and save to all disks
return collectNSaveNewFormatConfigs(referenceConfig, orderedDisks)
// generate and write new configs to all disks
return collectNSaveNewFormatConfigs(referenceConfig, orderedDisks,
dryRun)
}
// loadFormatXL - loads XL `format.json` and returns back properly
@@ -721,7 +582,7 @@ func loadFormatXL(bootstrapDisks []StorageAPI, readQuorum int) (disks []StorageA
var unformattedDisksFoundCnt = 0
var diskNotFoundCount = 0
var corruptedDisksFoundCnt = 0
formatConfigs := make([]*formatConfigV1, len(bootstrapDisks))
formats := make([]*formatXLV1, len(bootstrapDisks))
// Try to load `format.json` bootstrap disks.
for index, disk := range bootstrapDisks {
@@ -729,7 +590,7 @@ func loadFormatXL(bootstrapDisks []StorageAPI, readQuorum int) (disks []StorageA
diskNotFoundCount++
continue
}
var formatXL *formatConfigV1
var formatXL *formatXLV1
formatXL, err = loadFormat(disk)
if err != nil {
if err == errUnformattedDisk {
@@ -745,7 +606,7 @@ func loadFormatXL(bootstrapDisks []StorageAPI, readQuorum int) (disks []StorageA
return nil, err
}
// Save valid formats.
formatConfigs[index] = formatXL
formats[index] = formatXL
}
// If all disks indicate that 'format.json' is not available return 'errUnformattedDisk'.
@@ -760,59 +621,59 @@ func loadFormatXL(bootstrapDisks []StorageAPI, readQuorum int) (disks []StorageA
}
// Validate the format configs read are correct.
if err = checkFormatXL(formatConfigs); err != nil {
if err = checkFormatXL(formats); err != nil {
return nil, err
}
// Erasure code requires disks to be presented in the same
// order each time.
_, orderedDisks, err := reorderDisks(bootstrapDisks, formatConfigs,
_, orderedDisks, err := reorderDisks(bootstrapDisks, formats,
false)
return orderedDisks, err
}
func checkFormatXLValue(formatXL *formatConfigV1) error {
func checkFormatXLValue(formatXL *formatXLV1) error {
// Validate format version and format type.
if formatXL.Version != formatFileV1 {
if formatXL.Version != formatMetaVersionV1 {
return fmt.Errorf("Unsupported version of backend format [%s] found", formatXL.Version)
}
if formatXL.Format != formatBackendXL {
return fmt.Errorf("Unsupported backend format [%s] found", formatXL.Format)
}
if formatXL.XL.Version != "1" {
if formatXL.XL.Version != formatXLVersionV1 {
return fmt.Errorf("Unsupported XL backend format found [%s]", formatXL.XL.Version)
}
return nil
}
func checkFormatXLValues(formatConfigs []*formatConfigV1) (int, error) {
for i, formatXL := range formatConfigs {
func checkFormatXLValues(formats []*formatXLV1) (int, error) {
for i, formatXL := range formats {
if formatXL == nil {
continue
}
if err := checkFormatXLValue(formatXL); err != nil {
return i, err
}
if len(formatConfigs) != len(formatXL.XL.JBOD) {
if len(formats) != len(formatXL.XL.JBOD) {
return i, fmt.Errorf("Number of disks %d did not match the backend format %d",
len(formatConfigs), len(formatXL.XL.JBOD))
len(formats), len(formatXL.XL.JBOD))
}
}
return -1, nil
}
// checkFormatXL - verifies if format.json format is intact.
func checkFormatXL(formatConfigs []*formatConfigV1) error {
if _, err := checkFormatXLValues(formatConfigs); err != nil {
func checkFormatXL(formats []*formatXLV1) error {
if _, err := checkFormatXLValues(formats); err != nil {
return err
}
if err := checkJBODConsistency(formatConfigs); err != nil {
if err := checkJBODConsistency(formats); err != nil {
return err
}
return checkDisksConsistency(formatConfigs)
return checkDisksConsistency(formats)
}
// saveFormatXL - populates `format.json` on disks in its order.
func saveFormatXL(storageDisks []StorageAPI, formats []*formatConfigV1) error {
func saveFormatXL(storageDisks []StorageAPI, formats []*formatXLV1) error {
var errs = make([]error, len(storageDisks))
var wg = &sync.WaitGroup{}
// Write `format.json` to all disks.
@@ -821,7 +682,7 @@ func saveFormatXL(storageDisks []StorageAPI, formats []*formatConfigV1) error {
continue
}
wg.Add(1)
go func(index int, disk StorageAPI, format *formatConfigV1) {
go func(index int, disk StorageAPI, format *formatXLV1) {
defer wg.Done()
// Marshal and write to disk.
@@ -862,45 +723,37 @@ func saveFormatXL(storageDisks []StorageAPI, formats []*formatConfigV1) error {
return nil
}
// initFormatXL - save XL format configuration on all disks.
func initFormatXL(storageDisks []StorageAPI) (err error) {
// Initialize jbods.
var jbod = make([]string, len(storageDisks))
// Return a slice of format, to be used to format uninitialized disks.
func newFormatXLV1(diskCount int) []*formatXLV1 {
var jbod = make([]string, diskCount)
// Initialize formats.
var formats = make([]*formatConfigV1, len(storageDisks))
var formats = make([]*formatXLV1, diskCount)
// Initialize `format.json`.
for index, disk := range storageDisks {
if disk == nil {
continue
}
// Allocate format config.
formats[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: mustGetUUID(),
},
}
jbod[index] = formats[index].XL.Disk
for i := 0; i < diskCount; i++ {
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = mustGetUUID()
formats[i] = format
jbod[i] = formats[i].XL.Disk
}
// Update the jbod entries.
for index, disk := range storageDisks {
if disk == nil {
continue
}
// Save jbod.
formats[index].XL.JBOD = jbod
for i := 0; i < diskCount; i++ {
formats[i].XL.JBOD = jbod
}
return formats
}
// initFormatXL - save XL format configuration on all disks.
func initFormatXL(storageDisks []StorageAPI) (err error) {
// Initialize meta volume, if volume already exists ignores it.
if err := initMetaVolume(storageDisks); err != nil {
return fmt.Errorf("Unable to initialize '.minio.sys' meta volume, %s", err)
}
// Save formats `format.json` across all disks.
return saveFormatXL(storageDisks, formats)
return saveFormatXL(storageDisks, newFormatXLV1(len(storageDisks)))
}
@@ -18,54 +18,46 @@ package cmd
import (
"bytes"
"errors"
"os"
"path/filepath"
"testing"
errors2 "github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
"github.com/minio/minio/pkg/lock"
)
// generates a valid format.json for XL backend.
func genFormatXLValid() []*formatConfigV1 {
func genFormatXLValid() []*formatXLV1 {
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
return formatConfigs
}
// generates a invalid format.json version for XL backend.
func genFormatXLInvalidVersion() []*formatConfigV1 {
func genFormatXLInvalidVersion() []*formatXLV1 {
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
// Corrupt version numbers.
formatConfigs[0].Version = "2"
@@ -74,46 +66,42 @@ func genFormatXLInvalidVersion() []*formatConfigV1 {
}
// generates a invalid format.json version for XL backend.
func genFormatXLInvalidFormat() []*formatConfigV1 {
func genFormatXLInvalidFormat() []*formatXLV1 {
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
// Corrupt version numbers.
// Corrupt format.
formatConfigs[0].Format = "lx"
formatConfigs[3].Format = "lx"
return formatConfigs
}
// generates a invalid format.json version for XL backend.
func genFormatXLInvalidXLVersion() []*formatConfigV1 {
func genFormatXLInvalidXLVersion() []*formatXLV1 {
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
// Corrupt version numbers.
formatConfigs[0].XL.Version = "10"
@@ -121,51 +109,40 @@ func genFormatXLInvalidXLVersion() []*formatConfigV1 {
return formatConfigs
}
func genFormatFS() *formatConfigV1 {
return &formatConfigV1{
Version: formatFileV1,
Format: formatBackendFS,
}
}
// generates a invalid format.json version for XL backend.
func genFormatXLInvalidJBODCount() []*formatConfigV1 {
func genFormatXLInvalidJBODCount() []*formatXLV1 {
jbod := make([]string, 7)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
return formatConfigs
}
// generates a invalid format.json JBOD for XL backend.
func genFormatXLInvalidJBOD() []*formatConfigV1 {
func genFormatXLInvalidJBOD() []*formatXLV1 {
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
for index := range jbod {
jbod[index] = mustGetUUID()
@@ -177,22 +154,20 @@ func genFormatXLInvalidJBOD() []*formatConfigV1 {
}
// generates a invalid format.json Disk UUID for XL backend.
func genFormatXLInvalidDisks() []*formatConfigV1 {
func genFormatXLInvalidDisks() []*formatXLV1 {
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
// Make disk 5 and disk 8 have inconsistent disk uuid's.
formatConfigs[4].XL.Disk = mustGetUUID()
@@ -201,22 +176,20 @@ func genFormatXLInvalidDisks() []*formatConfigV1 {
}
// generates a invalid format.json Disk UUID in wrong order for XL backend.
func genFormatXLInvalidDisksOrder() []*formatConfigV1 {
func genFormatXLInvalidDisksOrder() []*formatXLV1 {
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
// Re order jbod for failure case.
var jbod1 = make([]string, 8)
@@ -298,7 +271,7 @@ func TestFormatXLHealFreshDisks(t *testing.T) {
formatConfigs, _ := loadAllFormats(storageDisks)
// Start healing disks
err = healFormatXLFreshDisks(storageDisks, formatConfigs)
err = healFormatXLFreshDisks(storageDisks, formatConfigs, false)
if err != nil {
t.Fatal("healing corrupted disk failed: ", err)
}
@@ -317,7 +290,7 @@ func TestFormatXLHealFreshDisks(t *testing.T) {
// a given disk to test healing a corrupted disk
func TestFormatXLHealCorruptedDisks(t *testing.T) {
// Create an instance of xl backend.
obj, fsDirs, err := prepareXL()
obj, fsDirs, err := prepareXL16()
if err != nil {
t.Fatal(err)
}
@@ -372,7 +345,7 @@ func TestFormatXLHealCorruptedDisks(t *testing.T) {
formatConfigs, _ := loadAllFormats(permutedStorageDisks)
// Start healing disks
err = healFormatXLCorruptedDisks(permutedStorageDisks, formatConfigs)
err = healFormatXLCorruptedDisks(permutedStorageDisks, formatConfigs, false)
if err != nil {
t.Fatal("healing corrupted disk failed: ", err)
}
@@ -391,7 +364,7 @@ func TestFormatXLHealCorruptedDisks(t *testing.T) {
// some of format.json
func TestFormatXLReorderByInspection(t *testing.T) {
// Create an instance of xl backend.
obj, fsDirs, err := prepareXL()
obj, fsDirs, err := prepareXL16()
if err != nil {
t.Fatal(err)
}
@@ -459,7 +432,7 @@ func TestFormatXLReorderByInspection(t *testing.T) {
// - invalid JBOD
// - invalid Disk uuid
func TestFormatXL(t *testing.T) {
formatInputCases := [][]*formatConfigV1{
formatInputCases := [][]*formatXLV1{
genFormatXLValid(),
genFormatXLInvalidVersion(),
genFormatXLInvalidFormat(),
@@ -470,7 +443,7 @@ func TestFormatXL(t *testing.T) {
genFormatXLInvalidDisksOrder(),
}
testCases := []struct {
formatConfigs []*formatConfigV1
formatConfigs []*formatXLV1
shouldPass bool
}{
{
@@ -525,22 +498,20 @@ func TestSavedUUIDOrder(t *testing.T) {
shouldPass bool
}, 8)
jbod := make([]string, 8)
formatConfigs := make([]*formatConfigV1, 8)
formatConfigs := make([]*formatXLV1, 8)
for index := range jbod {
jbod[index] = mustGetUUID()
uuidTestCases[index].uuid = jbod[index]
uuidTestCases[index].shouldPass = true
}
for index := range jbod {
formatConfigs[index] = &formatConfigV1{
Version: formatFileV1,
Format: formatBackendXL,
XL: &xlFormat{
Version: xlFormatBackendV1,
Disk: jbod[index],
JBOD: jbod,
},
}
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendXL
format.XL.Version = formatXLVersionV1
format.XL.Disk = jbod[index]
format.XL.JBOD = jbod
formatConfigs[index] = format
}
// Re order jbod for failure case.
var jbod1 = make([]string, 8)
@@ -653,172 +624,18 @@ func TestGenericFormatCheckXL(t *testing.T) {
t.Fatalf("Should fail here")
}
errs = []error{nil}
if err := genericFormatCheckXL([]*formatConfigV1{genFormatFS()}, errs); err == nil {
format := &formatXLV1{}
format.Version = formatMetaVersionV1
format.Format = formatBackendFS
if err := genericFormatCheckXL([]*formatXLV1{format}, errs); err == nil {
t.Fatalf("Should fail here")
}
errs = []error{errFaultyDisk}
if err := genericFormatCheckXL([]*formatConfigV1{genFormatFS()}, errs); err == nil {
if err := genericFormatCheckXL([]*formatXLV1{format}, errs); err == nil {
t.Fatalf("Should fail here")
}
}
// TestFSCheckFormatFSErr - test loadFormatFS loading older format.
func TestFSCheckFormatFSErr(t *testing.T) {
// Prepare for testing
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
defer os.RemoveAll(disk)
// Assign a new UUID.
uuid := mustGetUUID()
// Initialize meta volume, if volume already exists ignores it.
if err := initMetaVolumeFS(disk, uuid); err != nil {
t.Fatal(err)
}
testCases := []struct {
format *formatConfigV1
formatWriteErr error
formatCheckErr error
shouldPass bool
}{
{
format: &formatConfigV1{
Version: formatFileV1,
Format: formatBackendFS,
FS: &fsFormat{
Version: fsFormatBackendV1,
},
},
formatCheckErr: nil,
shouldPass: true,
},
{
format: &formatConfigV1{
Version: formatFileV1,
Format: formatBackendFS,
FS: &fsFormat{
Version: "10",
},
},
formatCheckErr: errors.New("Unknown backend FS format version '10'"),
shouldPass: false,
},
{
format: &formatConfigV1{
Version: formatFileV1,
Format: "garbage",
FS: &fsFormat{
Version: fsFormatBackendV1,
},
},
formatCheckErr: errors.New("FS backend format required. Found 'garbage'"),
},
{
format: &formatConfigV1{
Version: "-1",
Format: formatBackendFS,
FS: &fsFormat{
Version: fsFormatBackendV1,
},
},
formatCheckErr: errors.New("Unknown format file version '-1'"),
},
}
fsFormatPath := pathJoin(disk, minioMetaBucket, formatConfigFile)
for i, testCase := range testCases {
lk, err := lock.LockedOpenFile((fsFormatPath), os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
t.Fatal(err)
}
_, err = testCase.format.WriteTo(lk)
lk.Close()
if err != nil {
t.Fatalf("Test %d: Expected nil, got %s", i+1, err)
}
lk, err = lock.LockedOpenFile((fsFormatPath), os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
t.Fatal(err)
}
formatCfg := &formatConfigV1{}
_, err = formatCfg.ReadFrom(lk)
lk.Close()
if err != nil {
t.Fatal(err)
}
err = formatCfg.CheckFS()
if err != nil && testCase.shouldPass {
t.Errorf("Test %d: Should not fail with unexpected %s, expected nil", i+1, err)
}
if err == nil && !testCase.shouldPass {
t.Errorf("Test %d: Should fail with expected %s, got nil", i+1, testCase.formatCheckErr)
}
if err != nil && !testCase.shouldPass {
if errors2.Cause(err).Error() != testCase.formatCheckErr.Error() {
t.Errorf("Test %d: Should fail with expected %s, got %s", i+1, testCase.formatCheckErr, err)
}
}
}
}
// TestFSCheckFormatFS - test loadFormatFS with healty and faulty disks
func TestFSCheckFormatFS(t *testing.T) {
// Prepare for testing
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
defer os.RemoveAll(disk)
// Assign a new UUID.
uuid := mustGetUUID()
// Initialize meta volume, if volume already exists ignores it.
if err := initMetaVolumeFS(disk, uuid); err != nil {
t.Fatal(err)
}
fsFormatPath := pathJoin(disk, minioMetaBucket, formatConfigFile)
lk, err := lock.LockedOpenFile((fsFormatPath), os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
t.Fatal(err)
}
format := newFSFormatV1()
_, err = format.WriteTo(lk)
lk.Close()
if err != nil {
t.Fatal(err)
}
// Loading corrupted format file
file, err := os.OpenFile((fsFormatPath), os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
t.Fatal("Should not fail here", err)
}
file.Write([]byte{'b'})
file.Close()
lk, err = lock.LockedOpenFile((fsFormatPath), os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
t.Fatal(err)
}
format = &formatConfigV1{}
_, err = format.ReadFrom(lk)
lk.Close()
if err == nil {
t.Fatal("Should return an error here")
}
// Loading format file from disk not found.
os.RemoveAll(disk)
_, err = lock.LockedOpenFile((fsFormatPath), os.O_RDONLY, 0600)
if err != nil && !os.IsNotExist(err) {
t.Fatal("Should return 'format.json' does not exist, but got", err)
}
}
func TestLoadFormatXLErrs(t *testing.T) {
nDisks := 16
fsDirs, err := getRandomDisks(nDisks)
@@ -937,7 +754,7 @@ func TestHealFormatXLCorruptedDisksErrs(t *testing.T) {
xl := obj.(*xlObjects)
formatConfigs, _ := loadAllFormats(xl.storageDisks)
if err = healFormatXLCorruptedDisks(xl.storageDisks, formatConfigs); err != nil {
if err = healFormatXLCorruptedDisks(xl.storageDisks, formatConfigs, false); err != nil {
t.Fatal("Got an unexpected error: ", err)
}
@@ -960,7 +777,7 @@ func TestHealFormatXLCorruptedDisksErrs(t *testing.T) {
}
xl.storageDisks[0] = newNaughtyDisk(posixDisk, nil, errFaultyDisk)
formatConfigs, _ = loadAllFormats(xl.storageDisks)
if err = healFormatXLCorruptedDisks(xl.storageDisks, formatConfigs); err != errFaultyDisk {
if err = healFormatXLCorruptedDisks(xl.storageDisks, formatConfigs, false); err != errFaultyDisk {
t.Fatal("Got an unexpected error: ", err)
}
removeRoots(fsDirs)
@@ -982,7 +799,7 @@ func TestHealFormatXLCorruptedDisksErrs(t *testing.T) {
}
}
formatConfigs, _ = loadAllFormats(xl.storageDisks)
if err = healFormatXLCorruptedDisks(xl.storageDisks, formatConfigs); err == nil {
if err = healFormatXLCorruptedDisks(xl.storageDisks, formatConfigs, false); err == nil {
t.Fatal("Should get a json parsing error, ")
}
removeRoots(fsDirs)
@@ -1009,7 +826,7 @@ func TestHealFormatXLFreshDisksErrs(t *testing.T) {
}
xl := obj.(*xlObjects)
formatConfigs, _ := loadAllFormats(xl.storageDisks)
if err = healFormatXLFreshDisks(xl.storageDisks, formatConfigs); err != nil {
if err = healFormatXLFreshDisks(xl.storageDisks, formatConfigs, false); err != nil {
t.Fatal("Got an unexpected error: ", err)
}
removeRoots(fsDirs)
@@ -1031,7 +848,7 @@ func TestHealFormatXLFreshDisksErrs(t *testing.T) {
}
xl.storageDisks[0] = newNaughtyDisk(posixDisk, nil, errFaultyDisk)
formatConfigs, _ = loadAllFormats(xl.storageDisks)
if err = healFormatXLFreshDisks(xl.storageDisks, formatConfigs); err != errFaultyDisk {
if err = healFormatXLFreshDisks(xl.storageDisks, formatConfigs, false); err != errFaultyDisk {
t.Fatal("Got an unexpected error: ", err)
}
removeRoots(fsDirs)
@@ -1049,7 +866,7 @@ func TestHealFormatXLFreshDisksErrs(t *testing.T) {
xl = obj.(*xlObjects)
xl.storageDisks[0] = nil
formatConfigs, _ = loadAllFormats(xl.storageDisks)
if err = healFormatXLFreshDisks(xl.storageDisks, formatConfigs); err != nil {
if err = healFormatXLFreshDisks(xl.storageDisks, formatConfigs, false); err != nil {
t.Fatal("Got an unexpected error: ", err)
}
removeRoots(fsDirs)
-240
View File
@@ -1,240 +0,0 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"errors"
"io"
"os"
"reflect"
"sync"
"time"
)
// Error sent by appendParts go-routine when there are holes in parts.
// For ex. let's say client uploads part-2 before part-1 in which case we
// can not append and have to wait till part-1 is uploaded. Hence we return
// this error. Currently this error is not used in the caller.
var errPartsMissing = errors.New("required parts missing")
// Error sent when appendParts go-routine has waited long enough and timedout.
var errAppendPartsTimeout = errors.New("appendParts go-routine timeout")
// Timeout value for the appendParts go-routine.
var appendPartsTimeout = 24 * 60 * 60 * time.Second // 24 Hours.
// Holds a map of uploadID->appendParts go-routine
type backgroundAppend struct {
sync.Mutex
infoMap map[string]bgAppendPartsInfo
appendFile io.WriteCloser
}
// Input to the appendParts go-routine
type bgAppendPartsInput struct {
meta fsMetaV1 // list of parts that need to be appended
errCh chan error // error sent by appendParts go-routine
}
// Identifies an appendParts go-routine.
type bgAppendPartsInfo struct {
inputCh chan bgAppendPartsInput
timeoutCh chan struct{} // closed by appendParts go-routine when it timesout
abortCh chan struct{} // closed after abort of upload to end the appendParts go-routine
completeCh chan struct{} // closed after complete of upload to end the appendParts go-routine
}
// Called after a part is uploaded so that it can be appended in the background.
func (fs fsObjects) append(bucket, object, uploadID string, meta fsMetaV1) chan error {
fs.bgAppend.Lock()
info, ok := fs.bgAppend.infoMap[uploadID]
if !ok {
// Corresponding appendParts go-routine was not found, create a new one. Would happen when the first
// part of a multipart upload is uploaded.
inputCh := make(chan bgAppendPartsInput)
timeoutCh := make(chan struct{})
abortCh := make(chan struct{})
completeCh := make(chan struct{})
info = bgAppendPartsInfo{inputCh, timeoutCh, abortCh, completeCh}
fs.bgAppend.infoMap[uploadID] = info
go fs.appendParts(bucket, object, uploadID, info)
}
fs.bgAppend.Unlock()
errCh := make(chan error)
go func() {
// send input in a goroutine as send on the inputCh can block if appendParts go-routine
// is busy appending a part.
select {
case <-info.timeoutCh:
// This is to handle a rare race condition where we found info in b.infoMap
// but soon after that appendParts go-routine timed out.
errCh <- errAppendPartsTimeout
case info.inputCh <- bgAppendPartsInput{meta, errCh}:
}
}()
return errCh
}
// Called on complete-multipart-upload. Returns nil if the required parts have been appended.
func (fs *fsObjects) complete(bucket, object, uploadID string, meta fsMetaV1) error {
fs.bgAppend.Lock()
defer fs.bgAppend.Unlock()
info, ok := fs.bgAppend.infoMap[uploadID]
delete(fs.bgAppend.infoMap, uploadID)
if !ok {
return errPartsMissing
}
errCh := make(chan error)
select {
case <-info.timeoutCh:
// This is to handle a rare race condition where we found info in b.infoMap
// but soon after that appendParts go-routine timedouted out.
return errAppendPartsTimeout
case info.inputCh <- bgAppendPartsInput{meta, errCh}:
}
err := <-errCh
close(info.completeCh)
return err
}
// Called after complete-multipart-upload or abort-multipart-upload so that the appendParts go-routine is not left dangling.
func (fs fsObjects) abort(uploadID string) {
fs.bgAppend.Lock()
defer fs.bgAppend.Unlock()
info, ok := fs.bgAppend.infoMap[uploadID]
if !ok {
return
}
delete(fs.bgAppend.infoMap, uploadID)
info.abortCh <- struct{}{}
}
// This is run as a go-routine that appends the parts in the background.
func (fs fsObjects) appendParts(bucket, object, uploadID string, info bgAppendPartsInfo) {
appendPath := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID, uploadID)
// Holds the list of parts that is already appended to the "append" file.
appendMeta := fsMetaV1{}
// Allocate staging read buffer.
buf := make([]byte, readSizeV1)
for {
select {
case input := <-info.inputCh:
// We receive on this channel when new part gets uploaded or when complete-multipart sends
// a value on this channel to confirm if all the required parts are appended.
meta := input.meta
for {
// Append should be done such a way that if part-3 and part-2 is uploaded before part-1, we
// wait till part-1 is uploaded after which we append part-2 and part-3 as well in this for-loop.
part, appendNeeded := partToAppend(meta, appendMeta)
if !appendNeeded {
if reflect.DeepEqual(meta.Parts, appendMeta.Parts) {
// Sending nil is useful so that the complete-multipart-upload knows that
// all the required parts have been appended.
input.errCh <- nil
} else {
// Sending error is useful so that complete-multipart-upload can fall-back to
// its own append process.
input.errCh <- errPartsMissing
}
break
}
if err := fs.appendPart(bucket, object, uploadID, part, buf); err != nil {
fsRemoveFile(appendPath)
appendMeta.Parts = nil
input.errCh <- err
break
}
appendMeta.AddObjectPart(part.Number, part.Name, part.ETag, part.Size)
}
case <-info.abortCh:
// abort-multipart-upload closed abortCh to end the appendParts go-routine.
fsRemoveFile(appendPath)
// So that any racing PutObjectPart does not leave a dangling go-routine.
close(info.timeoutCh)
return
case <-info.completeCh:
// complete-multipart-upload closed completeCh to end the appendParts go-routine.
close(info.timeoutCh) // So that any racing PutObjectPart does not leave a dangling go-routine.
return
case <-time.After(appendPartsTimeout):
// Timeout the goroutine to garbage collect its resources. This would happen if the client initiates
// a multipart upload and does not complete/abort it.
fs.bgAppend.Lock()
delete(fs.bgAppend.infoMap, uploadID)
fs.bgAppend.Unlock()
// Delete the temporary append file as well.
fsRemoveFile(appendPath)
close(info.timeoutCh)
return
}
}
}
// Appends the "part" to the append-file inside "tmp/" that finally gets moved to the actual location
// upon complete-multipart-upload.
func (fs fsObjects) appendPart(bucket, object, uploadID string, part objectPartInfo, buf []byte) error {
partPath := pathJoin(fs.fsPath, minioMetaMultipartBucket, bucket, object, uploadID, part.Name)
var offset int64
// Read each file part to start writing to the temporary concatenated object.
file, size, err := fsOpenFile(partPath, offset)
if err != nil {
if err == errFileNotFound {
return errPartsMissing
}
return err
}
defer file.Close()
tmpObjPath := pathJoin(fs.fsPath, minioMetaTmpBucket, fs.fsUUID, uploadID)
// No need to hold a lock, this is a unique file and will be only written
// to one one process per uploadID per minio process.
wfile, err := os.OpenFile((tmpObjPath), os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0666)
if err != nil {
return err
}
defer wfile.Close()
// Fallocate more space as we concatenate.
if err = fsFAllocate(int(wfile.Fd()), 0, size); err != nil {
return err
}
_, err = io.CopyBuffer(wfile, file, buf)
return err
}
+10 -60
View File
@@ -1,5 +1,5 @@
/*
* Minio Cloud Storage, (C) 2016 Minio, Inc.
* Minio Cloud Storage, (C) 2016, 2017, 2018 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,13 +17,13 @@
package cmd
import (
"fmt"
"io"
"os"
pathutil "path"
"runtime"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/lock"
)
// Removes only the file at given path does not remove
@@ -91,24 +91,6 @@ func fsRemoveDir(dirPath string) (err error) {
return nil
}
// Creates a new directory, parent dir is also recursively created
// if it doesn't exist.
func fsMkdirAll(dirPath string) (err error) {
if dirPath == "" {
return errors.Trace(errInvalidArgument)
}
if err = checkPathLength(dirPath); err != nil {
return errors.Trace(err)
}
if err = os.MkdirAll(dirPath, 0777); err != nil {
return errors.Trace(err)
}
return nil
}
// Creates a new directory, parent dir should exist
// otherwise returns an error. If directory already
// exists returns an error. Windows long paths
@@ -277,7 +259,7 @@ func fsCreateFile(filePath string, reader io.Reader, buf []byte, fallocSize int6
return 0, errors.Trace(err)
}
if err := os.MkdirAll(pathutil.Dir(filePath), 0777); err != nil {
if err := mkdirAll(pathutil.Dir(filePath), 0777); err != nil {
return 0, errors.Trace(err)
}
@@ -285,7 +267,7 @@ func fsCreateFile(filePath string, reader io.Reader, buf []byte, fallocSize int6
return 0, errors.Trace(err)
}
writer, err := os.OpenFile((filePath), os.O_CREATE|os.O_WRONLY, 0666)
writer, err := lock.Open(filePath, os.O_CREATE|os.O_WRONLY, 0666)
if err != nil {
return 0, osErrToFSFileErr(err)
}
@@ -310,39 +292,10 @@ func fsCreateFile(filePath string, reader io.Reader, buf []byte, fallocSize int6
return 0, errors.Trace(err)
}
}
return bytesWritten, nil
}
// Removes uploadID at destination path.
func fsRemoveUploadIDPath(basePath, uploadIDPath string) error {
if basePath == "" || uploadIDPath == "" {
return errors.Trace(errInvalidArgument)
}
if err := checkPathLength(basePath); err != nil {
return errors.Trace(err)
}
if err := checkPathLength(uploadIDPath); err != nil {
return errors.Trace(err)
}
// List all the entries in uploadID.
entries, err := readDir(uploadIDPath)
if err != nil && err != errFileNotFound {
return errors.Trace(err)
}
// Delete all the entries obtained from previous readdir.
for _, entryPath := range entries {
err = fsDeleteFile(basePath, pathJoin(uploadIDPath, entryPath))
if err != nil && err != errFileNotFound {
return errors.Trace(err)
}
}
fsRemoveDir(uploadIDPath)
return nil
}
// fsFAllocate is similar to Fallocate but provides a convenient
// wrapper to handle various operating system specific errors.
func fsFAllocate(fd int, offset int64, len int64) (err error) {
@@ -374,19 +327,16 @@ func fsRenameFile(sourcePath, destPath string) error {
if err := checkPathLength(destPath); err != nil {
return errors.Trace(err)
}
// Verify if source path exists.
if _, err := os.Stat((sourcePath)); err != nil {
if _, err := os.Stat(sourcePath); err != nil {
return osErrToFSFileErr(err)
}
if err := os.MkdirAll(pathutil.Dir(destPath), 0777); err != nil {
return errors.Trace(err)
}
if err := os.Rename((sourcePath), (destPath)); err != nil {
if isSysErrCrossDevice(err) {
return errors.Trace(fmt.Errorf("%s (%s)->(%s)", errCrossDeviceLink, sourcePath, destPath))
}
if err := renameAll(sourcePath, destPath); err != nil {
return errors.Trace(err)
}
return nil
}
-22
View File
@@ -28,28 +28,6 @@ import (
"github.com/minio/minio/pkg/lock"
)
// Tests - fsMkdirAll()
func TestFSMkdirAll(t *testing.T) {
// create posix test setup
_, path, err := newPosixTestSetup()
if err != nil {
t.Fatalf("Unable to create posix test setup, %s", err)
}
defer os.RemoveAll(path)
if err = fsMkdirAll(""); errors.Cause(err) != errInvalidArgument {
t.Fatal("Unexpected error", err)
}
if err = fsMkdirAll(pathJoin(path, "my-obj-del-0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001")); errors.Cause(err) != errFileNameTooLong {
t.Fatal("Unexpected error", err)
}
if err = fsMkdirAll(pathJoin(path, "success-vol", "success-object")); err != nil {
t.Fatal("Unexpected error", err)
}
}
func TestFSRenameFile(t *testing.T) {
// create posix test setup
_, path, err := newPosixTestSetup()
+7 -178
View File
@@ -17,14 +17,11 @@
package cmd
import (
"encoding/json"
"io"
"io/ioutil"
"os"
pathutil "path"
"sort"
"strings"
"time"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/lock"
@@ -122,64 +119,21 @@ func (m fsMetaV1) ToObjectInfo(bucket, object string, fi os.FileInfo) ObjectInfo
// etag/md5Sum has already been extracted. We need to
// remove to avoid it from appearing as part of
// response headers. e.g, X-Minio-* or X-Amz-*.
objInfo.UserDefined = cleanMetaETag(m.Meta)
objInfo.UserDefined = cleanMetadata(m.Meta)
// Success..
return objInfo
}
// ObjectPartIndex - returns the index of matching object part number.
func (m fsMetaV1) ObjectPartIndex(partNumber int) (partIndex int) {
for i, part := range m.Parts {
if partNumber == part.Number {
partIndex = i
return partIndex
}
}
return -1
}
// AddObjectPart - add a new object part in order.
func (m *fsMetaV1) AddObjectPart(partNumber int, partName string, partETag string, partSize int64) {
partInfo := objectPartInfo{
Number: partNumber,
Name: partName,
ETag: partETag,
Size: partSize,
}
// Update part info if it already exists.
for i, part := range m.Parts {
if partNumber == part.Number {
m.Parts[i] = partInfo
return
}
}
// Proceed to include new part info.
m.Parts = append(m.Parts, partInfo)
// Parts in fsMeta should be in sorted order by part number.
sort.Sort(byObjectPartNumber(m.Parts))
}
func (m *fsMetaV1) WriteTo(lk *lock.LockedFile) (n int64, err error) {
var metadataBytes []byte
metadataBytes, err = json.Marshal(m)
if err = jsonSave(lk, m); err != nil {
return 0, err
}
fi, err := lk.Stat()
if err != nil {
return 0, errors.Trace(err)
return 0, err
}
if err = lk.Truncate(0); err != nil {
return 0, errors.Trace(err)
}
if _, err = lk.Write(metadataBytes); err != nil {
return 0, errors.Trace(err)
}
// Success.
return int64(len(metadataBytes)), nil
return fi.Size(), nil
}
func parseFSVersion(fsMetaBuf []byte) string {
@@ -204,21 +158,6 @@ func parseFSMetaMap(fsMetaBuf []byte) map[string]string {
return metaMap
}
func parseFSParts(fsMetaBuf []byte) []objectPartInfo {
// Parse the FS Parts.
partsResult := gjson.GetBytes(fsMetaBuf, "parts").Array()
partInfo := make([]objectPartInfo, len(partsResult))
for i, p := range partsResult {
info := objectPartInfo{}
info.Number = int(p.Get("number").Int())
info.Name = p.Get("name").String()
info.ETag = p.Get("etag").String()
info.Size = p.Get("size").Int()
partInfo[i] = info
}
return partInfo
}
func (m *fsMetaV1) ReadFrom(lk *lock.LockedFile) (n int64, err error) {
var fsMetaBuf []byte
fi, err := lk.Stat()
@@ -250,9 +189,6 @@ func (m *fsMetaV1) ReadFrom(lk *lock.LockedFile) (n int64, err error) {
// obtain metadata.
m.Meta = parseFSMetaMap(fsMetaBuf)
// obtain parts info list.
m.Parts = parseFSParts(fsMetaBuf)
// obtain minio release date.
m.Minio.Release = parseFSRelease(fsMetaBuf)
@@ -268,110 +204,3 @@ func newFSMetaV1() (fsMeta fsMetaV1) {
fsMeta.Minio.Release = ReleaseTag
return fsMeta
}
// Check if disk has already a valid format, holds a read lock and
// upon success returns it to the caller to be closed.
func checkLockedValidFormatFS(fsPath string) (*lock.RLockedFile, error) {
fsFormatPath := pathJoin(fsPath, minioMetaBucket, formatConfigFile)
rlk, err := lock.RLockedOpenFile((fsFormatPath))
if err != nil {
if os.IsNotExist(err) {
// If format.json not found then
// its an unformatted disk.
return nil, errors.Trace(errUnformattedDisk)
}
return nil, errors.Trace(err)
}
var format = &formatConfigV1{}
if err = format.LoadFormat(rlk.LockedFile); err != nil {
rlk.Close()
return nil, err
}
// Check format FS.
if err = format.CheckFS(); err != nil {
rlk.Close()
return nil, err
}
// Always return read lock here and should be closed by the caller.
return rlk, errors.Trace(err)
}
// Creates a new format.json if unformatted.
func createFormatFS(fsPath string) error {
fsFormatPath := pathJoin(fsPath, minioMetaBucket, formatConfigFile)
// Attempt a write lock on formatConfigFile `format.json`
// file stored in minioMetaBucket(.minio.sys) directory.
lk, err := lock.TryLockedOpenFile((fsFormatPath), os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
return errors.Trace(err)
}
// Close the locked file upon return.
defer lk.Close()
// Load format on disk, checks if we are unformatted
// writes the new format.json
var format = &formatConfigV1{}
err = format.LoadFormat(lk)
if errors.Cause(err) == errUnformattedDisk {
_, err = newFSFormat().WriteTo(lk)
return err
}
return err
}
func initFormatFS(fsPath string) (rlk *lock.RLockedFile, err error) {
// This loop validates format.json by holding a read lock and
// proceeds if disk unformatted to hold non-blocking WriteLock
// If for some reason non-blocking WriteLock fails and the error
// is lock.ErrAlreadyLocked i.e some other process is holding a
// lock we retry in the loop again.
for {
// Validate the `format.json` for expected values.
rlk, err = checkLockedValidFormatFS(fsPath)
switch {
case err == nil:
// Holding a read lock ensures that any write lock operation
// is blocked if attempted in-turn avoiding corruption on
// the backend disk.
return rlk, nil
case errors.Cause(err) == errUnformattedDisk:
if err = createFormatFS(fsPath); err != nil {
// Existing write locks detected.
if errors.Cause(err) == lock.ErrAlreadyLocked {
// Lock already present, sleep and attempt again.
time.Sleep(100 * time.Millisecond)
continue
}
// Unexpected error, return.
return nil, err
}
// Loop will continue to attempt a read-lock on `format.json`.
default:
// Unhandled error return.
return nil, err
}
}
}
// Return if the part info in uploadedParts and CompleteParts are same.
func isPartsSame(uploadedParts []objectPartInfo, CompleteParts []CompletePart) bool {
if len(uploadedParts) != len(CompleteParts) {
return false
}
for i := range CompleteParts {
if uploadedParts[i].Number != CompleteParts[i].PartNumber ||
uploadedParts[i].ETag != CompleteParts[i].ETag {
return false
}
}
return true
}
+395 -734
View File
File diff suppressed because it is too large Load Diff
+5 -83
View File
@@ -48,7 +48,7 @@ func TestFSCleanupMultipartUploadsInRoutine(t *testing.T) {
t.Fatal("Unexpected err: ", err)
}
go cleanupStaleMultipartUploads(20*time.Millisecond, 0, obj, fs.listMultipartUploadsCleanup, globalServiceDoneCh)
go fs.cleanupStaleMultipartUploads(20*time.Millisecond, 0, globalServiceDoneCh)
// Wait for 40ms such that - we have given enough time for
// cleanup routine to kick in.
@@ -66,71 +66,6 @@ func TestFSCleanupMultipartUploadsInRoutine(t *testing.T) {
}
}
// Tests cleanup of stale upload ids.
func TestFSCleanupMultipartUpload(t *testing.T) {
// Prepare for tests
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
defer os.RemoveAll(disk)
obj := initFSObjects(disk, t)
fs := obj.(*fsObjects)
// Close the multipart cleanup go-routine.
// In this test we are going to manually call
// the function which actually cleans the stale
// uploads.
globalServiceDoneCh <- struct{}{}
bucketName := "bucket"
objectName := "object"
obj.MakeBucketWithLocation(bucketName, "")
uploadID, err := obj.NewMultipartUpload(bucketName, objectName, nil)
if err != nil {
t.Fatal("Unexpected err: ", err)
}
if err = cleanupStaleMultipartUpload(bucketName, 0, obj, fs.listMultipartUploadsCleanup); err != nil {
t.Fatal("Unexpected err: ", err)
}
// Check if upload id was already purged.
if err = obj.AbortMultipartUpload(bucketName, objectName, uploadID); err != nil {
err = errors.Cause(err)
if _, ok := err.(InvalidUploadID); !ok {
t.Fatal("Unexpected err: ", err)
}
}
}
// TestFSWriteUploadJSON - tests for writeUploadJSON for FS
func TestFSWriteUploadJSON(t *testing.T) {
// Prepare for tests
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
defer os.RemoveAll(disk)
obj := initFSObjects(disk, t)
fs := obj.(*fsObjects)
bucketName := "bucket"
objectName := "object"
obj.MakeBucketWithLocation(bucketName, "")
_, err := obj.NewMultipartUpload(bucketName, objectName, nil)
if err != nil {
t.Fatal("Unexpected err: ", err)
}
// newMultipartUpload will fail.
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
_, err = obj.NewMultipartUpload(bucketName, objectName, nil)
if err != nil {
if _, ok := errors.Cause(err).(BucketNotFound); !ok {
t.Fatal("Unexpected err: ", err)
}
}
}
// TestNewMultipartUploadFaultyDisk - test NewMultipartUpload with faulty disks
func TestNewMultipartUploadFaultyDisk(t *testing.T) {
// Prepare for tests
@@ -215,12 +150,7 @@ func TestCompleteMultipartUploadFaultyDisk(t *testing.T) {
md5Hex := getMD5Hash(data)
if _, err := fs.PutObjectPart(bucketName, objectName, uploadID, 1, mustGetHashReader(t, bytes.NewReader(data), 5, md5Hex, "")); err != nil {
t.Fatal("Unexpected error ", err)
}
parts := []CompletePart{{PartNumber: 1, ETag: md5Hex}}
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if _, err := fs.CompleteMultipartUpload(bucketName, objectName, uploadID, parts); err != nil {
if !isSameType(errors.Cause(err), BucketNotFound{}) {
@@ -229,7 +159,7 @@ func TestCompleteMultipartUploadFaultyDisk(t *testing.T) {
}
}
// TestCompleteMultipartUploadFaultyDisk - test CompleteMultipartUpload with faulty disks
// TestCompleteMultipartUpload - test CompleteMultipartUpload
func TestCompleteMultipartUpload(t *testing.T) {
// Prepare for tests
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
@@ -263,7 +193,7 @@ func TestCompleteMultipartUpload(t *testing.T) {
}
}
// TestCompleteMultipartUploadFaultyDisk - test CompleteMultipartUpload with faulty disks
// TestCompleteMultipartUpload - test CompleteMultipartUpload
func TestAbortMultipartUpload(t *testing.T) {
// Prepare for tests
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
@@ -289,7 +219,7 @@ func TestAbortMultipartUpload(t *testing.T) {
if _, err := fs.PutObjectPart(bucketName, objectName, uploadID, 1, mustGetHashReader(t, bytes.NewReader(data), 5, md5Hex, "")); err != nil {
t.Fatal("Unexpected error ", err)
}
time.Sleep(time.Second) // Without Sleep on windows, the fs.AbortMultipartUpload() fails with "The process cannot access the file because it is being used by another process."
if err := fs.AbortMultipartUpload(bucketName, objectName, uploadID); err != nil {
t.Fatal("Unexpected error ", err)
}
@@ -306,24 +236,16 @@ func TestListMultipartUploadsFaultyDisk(t *testing.T) {
fs := obj.(*fsObjects)
bucketName := "bucket"
objectName := "object"
data := []byte("12345")
if err := obj.MakeBucketWithLocation(bucketName, ""); err != nil {
t.Fatal("Cannot create bucket, err: ", err)
}
uploadID, err := fs.NewMultipartUpload(bucketName, objectName, map[string]string{"X-Amz-Meta-xid": "3f"})
_, err := fs.NewMultipartUpload(bucketName, objectName, map[string]string{"X-Amz-Meta-xid": "3f"})
if err != nil {
t.Fatal("Unexpected error ", err)
}
md5Hex := getMD5Hash(data)
sha256sum := ""
if _, err := fs.PutObjectPart(bucketName, objectName, uploadID, 1, mustGetHashReader(t, bytes.NewReader(data), 5, md5Hex, sha256sum)); err != nil {
t.Fatal("Unexpected error ", err)
}
fs.fsPath = filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
if _, err := fs.ListMultipartUploads(bucketName, objectName, "", "", "", 1000); err != nil {
if !isSameType(errors.Cause(err), BucketNotFound{}) {
+145 -69
View File
@@ -25,11 +25,14 @@ import (
"path"
"path/filepath"
"sort"
"sync"
"syscall"
"time"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
"github.com/minio/minio/pkg/lock"
"github.com/minio/minio/pkg/madmin"
)
// fsObjects - Implements fs object layer.
@@ -50,8 +53,18 @@ type fsObjects struct {
// ListObjects pool management.
listPool *treeWalkPool
// To manage the appendRoutine go0routines
bgAppend *backgroundAppend
appendFileMap map[string]*fsAppendFile
appendFileMapMu sync.Mutex
// To manage the appendRoutine go-routines
nsMutex *nsLockMap
}
// Represents the background append file.
type fsAppendFile struct {
sync.Mutex
parts []PartInfo // List of parts appended.
filePath string // Absolute path of the file in the temp location.
}
// Initializes meta volume on all the fs path.
@@ -134,10 +147,9 @@ func newFSObjectLayer(fsPath string) (ObjectLayer, error) {
rwPool: &fsIOPool{
readersMap: make(map[string]*lock.RLockedFile),
},
listPool: newTreeWalkPool(globalLookupTimeout),
bgAppend: &backgroundAppend{
infoMap: make(map[string]bgAppendPartsInfo),
},
nsMutex: newNSLock(false),
listPool: newTreeWalkPool(globalLookupTimeout),
appendFileMap: make(map[string]*fsAppendFile),
}
// Once the filesystem has initialized hold the read lock for
@@ -156,15 +168,13 @@ func newFSObjectLayer(fsPath string) (ObjectLayer, error) {
return nil, fmt.Errorf("Unable to initialize event notification. %s", err)
}
// Start background process to cleanup old multipart objects in `.minio.sys`.
go cleanupStaleMultipartUploads(multipartCleanupInterval, multipartExpiry, fs, fs.listMultipartUploadsCleanup, globalServiceDoneCh)
go fs.cleanupStaleMultipartUploads(multipartCleanupInterval, multipartExpiry, globalServiceDoneCh)
// Return successfully initialized object layer.
return fs, nil
}
// Should be called when process shuts down.
func (fs fsObjects) Shutdown() error {
func (fs *fsObjects) Shutdown() error {
fs.fsFormatRlk.Close()
// Cleanup and delete tmp uuid.
@@ -172,7 +182,7 @@ func (fs fsObjects) Shutdown() error {
}
// StorageInfo - returns underlying storage statistics.
func (fs fsObjects) StorageInfo() StorageInfo {
func (fs *fsObjects) StorageInfo() StorageInfo {
info, err := getDiskInfo((fs.fsPath))
errorIf(err, "Unable to get disk info %#v", fs.fsPath)
storageInfo := StorageInfo{
@@ -183,12 +193,24 @@ func (fs fsObjects) StorageInfo() StorageInfo {
return storageInfo
}
// Locking operations
// List namespace locks held in object layer
func (fs *fsObjects) ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
return []VolumeLockInfo{}, NotImplemented{}
}
// Clear namespace locks held in object layer
func (fs *fsObjects) ClearLocks([]VolumeLockInfo) error {
return NotImplemented{}
}
/// Bucket operations
// getBucketDir - will convert incoming bucket names to
// corresponding valid bucket names on the backend in a platform
// compatible way for all operating systems.
func (fs fsObjects) getBucketDir(bucket string) (string, error) {
func (fs *fsObjects) getBucketDir(bucket string) (string, error) {
// Verify if bucket is valid.
if !IsValidBucketName(bucket) {
return "", errors.Trace(BucketNameInvalid{Bucket: bucket})
@@ -198,7 +220,7 @@ func (fs fsObjects) getBucketDir(bucket string) (string, error) {
return bucketDir, nil
}
func (fs fsObjects) statBucketDir(bucket string) (os.FileInfo, error) {
func (fs *fsObjects) statBucketDir(bucket string) (os.FileInfo, error) {
bucketDir, err := fs.getBucketDir(bucket)
if err != nil {
return nil, err
@@ -212,7 +234,12 @@ func (fs fsObjects) statBucketDir(bucket string) (os.FileInfo, error) {
// MakeBucket - create a new bucket, returns if it
// already exists.
func (fs fsObjects) MakeBucketWithLocation(bucket, location string) error {
func (fs *fsObjects) MakeBucketWithLocation(bucket, location string) error {
bucketLock := fs.nsMutex.NewNSLock(bucket, "")
if err := bucketLock.GetLock(globalObjectTimeout); err != nil {
return err
}
defer bucketLock.Unlock()
bucketDir, err := fs.getBucketDir(bucket)
if err != nil {
return toObjectErr(err, bucket)
@@ -226,7 +253,7 @@ func (fs fsObjects) MakeBucketWithLocation(bucket, location string) error {
}
// GetBucketInfo - fetch bucket metadata info.
func (fs fsObjects) GetBucketInfo(bucket string) (bi BucketInfo, e error) {
func (fs *fsObjects) GetBucketInfo(bucket string) (bi BucketInfo, e error) {
st, err := fs.statBucketDir(bucket)
if err != nil {
return bi, toObjectErr(err, bucket)
@@ -241,7 +268,7 @@ func (fs fsObjects) GetBucketInfo(bucket string) (bi BucketInfo, e error) {
}
// ListBuckets - list all s3 compatible buckets (directories) at fsPath.
func (fs fsObjects) ListBuckets() ([]BucketInfo, error) {
func (fs *fsObjects) ListBuckets() ([]BucketInfo, error) {
if err := checkPathLength(fs.fsPath); err != nil {
return nil, errors.Trace(err)
}
@@ -282,7 +309,7 @@ func (fs fsObjects) ListBuckets() ([]BucketInfo, error) {
// DeleteBucket - delete a bucket and all the metadata associated
// with the bucket including pending multipart, object metadata.
func (fs fsObjects) DeleteBucket(bucket string) error {
func (fs *fsObjects) DeleteBucket(bucket string) error {
bucketDir, err := fs.getBucketDir(bucket)
if err != nil {
return toObjectErr(err, bucket)
@@ -313,7 +340,29 @@ func (fs fsObjects) DeleteBucket(bucket string) error {
// CopyObject - copy object source object to destination object.
// if source object and destination object are same we only
// update metadata.
func (fs fsObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string, metadata map[string]string) (oi ObjectInfo, e error) {
func (fs *fsObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string, metadata map[string]string, srcEtag string) (oi ObjectInfo, e error) {
cpSrcDstSame := srcBucket == dstBucket && srcObject == dstObject
// Hold write lock on destination since in both cases
// - if source and destination are same
// - if source and destination are different
// it is the sole mutating state.
objectDWLock := fs.nsMutex.NewNSLock(dstBucket, dstObject)
if err := objectDWLock.GetLock(globalObjectTimeout); err != nil {
return oi, err
}
defer objectDWLock.Unlock()
// if source and destination are different, we have to hold
// additional read lock as well to protect against writes on
// source.
if !cpSrcDstSame {
// Hold read locks on source object only if we are
// going to read data from source object.
objectSRLock := fs.nsMutex.NewNSLock(srcBucket, srcObject)
if err := objectSRLock.GetRLock(globalObjectTimeout); err != nil {
return oi, err
}
defer objectSRLock.RUnlock()
}
if _, err := fs.statBucketDir(srcBucket); err != nil {
return oi, toObjectErr(err, srcBucket)
}
@@ -323,6 +372,15 @@ func (fs fsObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string
if err != nil {
return oi, toObjectErr(err, srcBucket, srcObject)
}
if srcEtag != "" {
etag, perr := fs.getObjectETag(srcBucket, srcObject)
if perr != nil {
return oi, toObjectErr(perr, srcBucket, srcObject)
}
if etag != srcEtag {
return oi, toObjectErr(errors.Trace(InvalidETag{}), srcBucket, srcObject)
}
}
// Check if this request is only metadata update.
cpMetadataOnly := isStringEqual(pathJoin(srcBucket, srcObject), pathJoin(dstBucket, dstObject))
@@ -355,7 +413,7 @@ func (fs fsObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string
go func() {
var startOffset int64 // Read the whole file.
if gerr := fs.GetObject(srcBucket, srcObject, startOffset, length, pipeWriter); gerr != nil {
if gerr := fs.getObject(srcBucket, srcObject, startOffset, length, pipeWriter, ""); gerr != nil {
errorIf(gerr, "Unable to read %s/%s.", srcBucket, srcObject)
pipeWriter.CloseWithError(gerr)
return
@@ -368,7 +426,7 @@ func (fs fsObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string
return oi, toObjectErr(err, dstBucket, dstObject)
}
objInfo, err := fs.PutObject(dstBucket, dstObject, hashReader, metadata)
objInfo, err := fs.putObject(dstBucket, dstObject, hashReader, metadata)
if err != nil {
return oi, toObjectErr(err, dstBucket, dstObject)
}
@@ -385,11 +443,22 @@ func (fs fsObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (fs fsObjects) GetObject(bucket, object string, offset int64, length int64, writer io.Writer) (err error) {
if err = checkBucketAndObjectNamesFS(bucket, object); err != nil {
func (fs *fsObjects) GetObject(bucket, object string, offset int64, length int64, writer io.Writer, etag string) (err error) {
if err = checkGetObjArgs(bucket, object); err != nil {
return err
}
// Lock the object before reading.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
if err := objectLock.GetRLock(globalObjectTimeout); err != nil {
return err
}
defer objectLock.RUnlock()
return fs.getObject(bucket, object, offset, length, writer, etag)
}
// getObject - wrapper for GetObject
func (fs *fsObjects) getObject(bucket, object string, offset int64, length int64, writer io.Writer, etag string) (err error) {
if _, err = fs.statBucketDir(bucket); err != nil {
return toObjectErr(err, bucket)
}
@@ -419,6 +488,15 @@ func (fs fsObjects) GetObject(bucket, object string, offset int64, length int64,
defer fs.rwPool.Close(fsMetaPath)
}
if etag != "" {
objEtag, perr := fs.getObjectETag(bucket, object)
if perr != nil {
return toObjectErr(errors.Trace(perr), bucket, object)
}
if objEtag != etag {
return toObjectErr(errors.Trace(InvalidETag{}), bucket, object)
}
}
// Read the object, doesn't exist returns an s3 compatible error.
fsObjPath := pathJoin(fs.fsPath, bucket, object)
reader, size, err := fsOpenFile(fsObjPath, offset)
@@ -451,7 +529,7 @@ func (fs fsObjects) GetObject(bucket, object string, offset int64, length int64,
}
// getObjectInfo - wrapper for reading object metadata and constructs ObjectInfo.
func (fs fsObjects) getObjectInfo(bucket, object string) (oi ObjectInfo, e error) {
func (fs *fsObjects) getObjectInfo(bucket, object string) (oi ObjectInfo, e error) {
fsMeta := fsMetaV1{}
fi, err := fsStatDir(pathJoin(fs.fsPath, bucket, object))
if err != nil && errors.Cause(err) != errFileAccessDenied {
@@ -498,25 +576,16 @@ func (fs fsObjects) getObjectInfo(bucket, object string) (oi ObjectInfo, e error
return fsMeta.ToObjectInfo(bucket, object, fi), nil
}
// Checks bucket and object name validity, returns nil if both are valid.
func checkBucketAndObjectNamesFS(bucket, object string) error {
// Verify if bucket is valid.
if !IsValidBucketName(bucket) {
return errors.Trace(BucketNameInvalid{Bucket: bucket})
}
// Verify if object is valid.
if len(object) == 0 {
return errors.Trace(ObjectNameInvalid{Bucket: bucket, Object: object})
}
if !IsValidObjectPrefix(object) {
return errors.Trace(ObjectNameInvalid{Bucket: bucket, Object: object})
}
return nil
}
// GetObjectInfo - reads object metadata and replies back ObjectInfo.
func (fs fsObjects) GetObjectInfo(bucket, object string) (oi ObjectInfo, e error) {
if err := checkBucketAndObjectNamesFS(bucket, object); err != nil {
func (fs *fsObjects) GetObjectInfo(bucket, object string) (oi ObjectInfo, e error) {
// Lock the object before reading.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
if err := objectLock.GetRLock(globalObjectTimeout); err != nil {
return oi, err
}
defer objectLock.RUnlock()
if err := checkGetObjArgs(bucket, object); err != nil {
return oi, err
}
@@ -530,7 +599,7 @@ func (fs fsObjects) GetObjectInfo(bucket, object string) (oi ObjectInfo, e error
// This function does the following check, suppose
// object is "a/b/c/d", stat makes sure that objects ""a/b/c""
// "a/b" and "a" do not exist.
func (fs fsObjects) parentDirIsObject(bucket, parent string) bool {
func (fs *fsObjects) parentDirIsObject(bucket, parent string) bool {
var isParentDirObject func(string) bool
isParentDirObject = func(p string) bool {
if p == "." || p == "/" {
@@ -551,12 +620,22 @@ func (fs fsObjects) parentDirIsObject(bucket, parent string) bool {
// until EOF, writes data directly to configured filesystem path.
// Additionally writes `fs.json` which carries the necessary metadata
// for future object operations.
func (fs fsObjects) PutObject(bucket string, object string, data *hash.Reader, metadata map[string]string) (objInfo ObjectInfo, retErr error) {
func (fs *fsObjects) PutObject(bucket string, object string, data *hash.Reader, metadata map[string]string) (objInfo ObjectInfo, retErr error) {
// Lock the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
if err := objectLock.GetLock(globalObjectTimeout); err != nil {
return objInfo, err
}
defer objectLock.Unlock()
return fs.putObject(bucket, object, data, metadata)
}
// putObject - wrapper for PutObject
func (fs *fsObjects) putObject(bucket string, object string, data *hash.Reader, metadata map[string]string) (objInfo ObjectInfo, retErr error) {
// No metadata is set, allocate a new one.
if metadata == nil {
metadata = make(map[string]string)
}
var err error
// Validate if bucket name is valid and exists.
@@ -575,7 +654,7 @@ func (fs fsObjects) PutObject(bucket string, object string, data *hash.Reader, m
if fs.parentDirIsObject(bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errors.Trace(errFileAccessDenied), bucket, object)
}
if err = fsMkdirAll(pathJoin(fs.fsPath, bucket, object)); err != nil {
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
var fi os.FileInfo
@@ -634,7 +713,6 @@ func (fs fsObjects) PutObject(bucket string, object string, data *hash.Reader, m
bytesWritten, err := fsCreateFile(fsTmpObjPath, data, buf, data.Size())
if err != nil {
fsRemoveFile(fsTmpObjPath)
errorIf(err, "Failed to create object %s/%s", bucket, object)
return ObjectInfo{}, toObjectErr(err, bucket, object)
}
@@ -677,8 +755,15 @@ func (fs fsObjects) PutObject(bucket string, object string, data *hash.Reader, m
// DeleteObject - deletes an object from a bucket, this operation is destructive
// and there are no rollbacks supported.
func (fs fsObjects) DeleteObject(bucket, object string) error {
if err := checkBucketAndObjectNamesFS(bucket, object); err != nil {
func (fs *fsObjects) DeleteObject(bucket, object string) error {
// Acquire a write lock before deleting the object.
objectLock := fs.nsMutex.NewNSLock(bucket, object)
if err := objectLock.GetLock(globalOperationTimeout); err != nil {
return err
}
defer objectLock.Unlock()
if err := checkDelObjArgs(bucket, object); err != nil {
return err
}
@@ -714,16 +799,10 @@ func (fs fsObjects) DeleteObject(bucket, object string) error {
return nil
}
// list of all errors that can be ignored in tree walk operation in FS
var fsTreeWalkIgnoredErrs = append(baseIgnoredErrs, []error{
errFileNotFound,
errVolumeNotFound,
}...)
// Returns function "listDir" of the type listDirFunc.
// isLeaf - is used by listDir function to check if an entry
// is a leaf or non-leaf entry.
func (fs fsObjects) listDirFactory(isLeaf isLeafFunc) listDirFunc {
func (fs *fsObjects) listDirFactory(isLeaf isLeafFunc) listDirFunc {
// listDir - lists all the entries at a given prefix and given entry in the prefix.
listDir := func(bucket, prefixDir, prefixEntry string) (entries []string, delayIsLeaf bool, err error) {
entries, err = readDir(pathJoin(fs.fsPath, bucket, prefixDir))
@@ -740,7 +819,7 @@ func (fs fsObjects) listDirFactory(isLeaf isLeafFunc) listDirFunc {
// getObjectETag is a helper function, which returns only the md5sum
// of the file on the disk.
func (fs fsObjects) getObjectETag(bucket, entry string) (string, error) {
func (fs *fsObjects) getObjectETag(bucket, entry string) (string, error) {
fsMetaPath := pathJoin(fs.fsPath, minioMetaBucket, bucketMetaPrefix, bucket, entry, fsMetaJSONFile)
// Read `fs.json` to perhaps contend with
@@ -789,7 +868,7 @@ func (fs fsObjects) getObjectETag(bucket, entry string) (string, error) {
// ListObjects - list all objects at prefix upto maxKeys., optionally delimited by '/'. Maintains the list pool
// state for future re-entrant list requests.
func (fs fsObjects) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
func (fs *fsObjects) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
if err := checkListObjsArgs(bucket, prefix, marker, delimiter, fs); err != nil {
return loi, err
}
@@ -826,7 +905,7 @@ func (fs fsObjects) ListObjects(bucket, prefix, marker, delimiter string, maxKey
// Convert entry to ObjectInfo
entryToObjectInfo := func(entry string) (objInfo ObjectInfo, err error) {
// Protect the entry from concurrent deletes, or renames.
objectLock := globalNSMutex.NewNSLock(bucket, entry)
objectLock := fs.nsMutex.NewNSLock(bucket, entry)
if err = objectLock.GetRLock(globalListingTimeout); err != nil {
return ObjectInfo{}, err
}
@@ -943,26 +1022,23 @@ func (fs fsObjects) ListObjects(bucket, prefix, marker, delimiter string, maxKey
}
// HealObject - no-op for fs. Valid only for XL.
func (fs fsObjects) HealObject(bucket, object string) (int, int, error) {
return 0, 0, errors.Trace(NotImplemented{})
func (fs *fsObjects) HealObject(bucket, object string, dryRun bool) (
res madmin.HealResultItem, err error) {
return res, errors.Trace(NotImplemented{})
}
// HealBucket - no-op for fs, Valid only for XL.
func (fs fsObjects) HealBucket(bucket string) error {
return errors.Trace(NotImplemented{})
func (fs *fsObjects) HealBucket(bucket string, dryRun bool) ([]madmin.HealResultItem,
error) {
return nil, errors.Trace(NotImplemented{})
}
// ListObjectsHeal - list all objects to be healed. Valid only for XL
func (fs fsObjects) ListObjectsHeal(bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
func (fs *fsObjects) ListObjectsHeal(bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
return loi, errors.Trace(NotImplemented{})
}
// ListBucketsHeal - list all buckets to be healed. Valid only for XL
func (fs fsObjects) ListBucketsHeal() ([]BucketInfo, error) {
func (fs *fsObjects) ListBucketsHeal() ([]BucketInfo, error) {
return []BucketInfo{}, errors.Trace(NotImplemented{})
}
func (fs fsObjects) ListUploadsHeal(bucket, prefix, marker, uploadIDMarker,
delimiter string, maxUploads int) (lmi ListMultipartsInfo, e error) {
return lmi, errors.Trace(NotImplemented{})
}
+1 -1
View File
@@ -400,7 +400,7 @@ func TestFSHealObject(t *testing.T) {
defer os.RemoveAll(disk)
obj := initFSObjects(disk, t)
_, _, err := obj.HealObject("bucket", "object")
_, err := obj.HealObject("bucket", "object", false)
if err == nil || !isSameType(errors.Cause(err), NotImplemented{}) {
t.Fatalf("Heal Object should return NotImplemented error ")
}
+1 -25
View File
@@ -81,7 +81,6 @@ func (api gatewayAPIHandlers) GetObjectHandler(w http.ResponseWriter, r *http.Re
}
objInfo, err := getObjectInfo(bucket, object)
if err != nil {
errorIf(err, "Unable to fetch object info.")
apiErr := toAPIErrorCode(err)
if apiErr == ErrNoSuchKey {
apiErr = errAllowableObjectNotFound(bucket, r)
@@ -129,7 +128,7 @@ func (api gatewayAPIHandlers) GetObjectHandler(w http.ResponseWriter, r *http.Re
setHeadGetRespHeaders(w, r.URL.Query())
httpWriter := ioutil.WriteOnClose(w)
// Reads the object at startOffset and writes to mw.
if err = getObject(bucket, object, startOffset, length, httpWriter); err != nil {
if err = getObject(bucket, object, startOffset, length, httpWriter, objInfo.ETag); err != nil {
errorIf(err, "Unable to write to client.")
if !httpWriter.HasWritten() {
// Error response only if no data has been written to client yet. i.e if
@@ -199,7 +198,6 @@ func (api gatewayAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Re
// Get Content-Md5 sent by client and verify if valid
md5Bytes, err := checkValidMD5(r.Header.Get("Content-Md5"))
if err != nil {
errorIf(err, "Unable to validate content-md5 format.")
writeErrorResponse(w, ErrInvalidDigest, r.URL)
return
}
@@ -211,7 +209,6 @@ func (api gatewayAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Re
sizeStr := r.Header.Get("x-amz-decoded-content-length")
size, err = strconv.ParseInt(sizeStr, 10, 64)
if err != nil {
errorIf(err, "Unable to parse `x-amz-decoded-content-length` into its integer value", sizeStr)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -230,7 +227,6 @@ func (api gatewayAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Re
// Extract metadata to be saved from incoming HTTP header.
metadata, err := extractMetadataFromHeader(r.Header)
if err != nil {
errorIf(err, "found invalid http request header")
writeErrorResponse(w, ErrInternalError, r.URL)
return
}
@@ -254,14 +250,6 @@ func (api gatewayAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Re
}
}
// Lock the object.
objectLock := globalNSMutex.NewNSLock(bucket, object)
if objectLock.GetLock(globalOperationTimeout) != nil {
writeErrorResponse(w, ErrOperationTimedOut, r.URL)
return
}
defer objectLock.Unlock()
var (
// Make sure we hex encode md5sum here.
md5hex = hex.EncodeToString(md5Bytes)
@@ -385,7 +373,6 @@ func (api gatewayAPIHandlers) HeadObjectHandler(w http.ResponseWriter, r *http.R
}
objInfo, err := getObjectInfo(bucket, object)
if err != nil {
errorIf(err, "Unable to fetch object info.")
apiErr := toAPIErrorCode(err)
if apiErr == ErrNoSuchKey {
apiErr = errAllowableObjectNotFound(bucket, r)
@@ -448,7 +435,6 @@ func (api gatewayAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *h
// Before proceeding validate if bucket exists.
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -511,7 +497,6 @@ func (api gatewayAPIHandlers) DeleteBucketPolicyHandler(w http.ResponseWriter, r
// Before proceeding validate if bucket exists.
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -545,21 +530,18 @@ func (api gatewayAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *h
// Before proceeding validate if bucket exists.
_, err := objAPI.GetBucketInfo(bucket)
if err != nil {
errorIf(err, "Unable to find bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
bp, err := objAPI.GetBucketPolicies(bucket)
if err != nil {
errorIf(err, "Unable to read bucket policy.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
policyBytes, err := json.Marshal(bp)
if err != nil {
errorIf(err, "Unable to read bucket policy.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -631,7 +613,6 @@ func (api gatewayAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Re
// Proceed to creating a bucket.
err := objectAPI.MakeBucketWithLocation(bucket, location)
if err != nil {
errorIf(err, "Unable to create a bucket.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -661,7 +642,6 @@ func (api gatewayAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http
// Attempt to delete bucket.
if err := objectAPI.DeleteBucket(bucket); err != nil {
errorIf(err, "Unable to delete a bucket.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -733,7 +713,6 @@ func (api gatewayAPIHandlers) ListObjectsV1Handler(w http.ResponseWriter, r *htt
// marshalled into S3 compatible XML header.
listObjectsInfo, err := listObjects(bucket, prefix, marker, delimiter, maxKeys)
if err != nil {
errorIf(err, "Unable to list objects.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -813,7 +792,6 @@ func (api gatewayAPIHandlers) ListObjectsV2Handler(w http.ResponseWriter, r *htt
// serialized as XML and sent as S3 compatible response body.
listObjectsV2Info, err := listObjectsV2(bucket, prefix, token, delimiter, maxKeys, fetchOwner, startAfter)
if err != nil {
errorIf(err, "Unable to list objects. Args to listObjectsV2 are bucket=%s, prefix=%s, token=%s, delimiter=%s", bucket, prefix, token, delimiter)
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
@@ -871,7 +849,6 @@ func (api gatewayAPIHandlers) HeadBucketHandler(w http.ResponseWriter, r *http.R
}
if _, err := getBucketInfo(bucket); err != nil {
errorIf(err, "Unable to fetch bucket info.")
writeErrorResponseHeadersOnly(w, toAPIErrorCode(err))
return
}
@@ -927,7 +904,6 @@ func (api gatewayAPIHandlers) GetBucketLocationHandler(w http.ResponseWriter, r
}
if _, err := getBucketInfo(bucket); err != nil {
errorIf(err, "Unable to fetch bucket info.")
writeErrorResponse(w, toAPIErrorCode(err), r.URL)
return
}
+7
View File
@@ -112,6 +112,13 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
cli.ShowCommandHelpAndExit(ctx, gatewayName, 1)
}
// Get "json" flag from command line argument and
// enable json and quite modes if jason flag is turned on.
jsonFlag := ctx.Bool("json") || ctx.GlobalBool("json")
if jsonFlag {
log.EnableJSON()
}
// Get quiet flag from command line argument.
quietFlag := ctx.Bool("quiet") || ctx.GlobalBool("quiet")
if quietFlag {
+1 -1
View File
@@ -45,7 +45,7 @@ type Gateway interface {
type GatewayLayer interface {
ObjectLayer
AnonGetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) (err error)
AnonGetObject(bucket, object string, startOffset int64, length int64, writer io.Writer, etag string) (err error)
AnonGetObjectInfo(bucket, object string) (objInfo ObjectInfo, err error)
AnonPutObject(bucket string, object string, data *hash.Reader, metadata map[string]string) (ObjectInfo, error)
+4 -2
View File
@@ -53,6 +53,8 @@ func printGatewayCommonMsg(apiEndpoints []string) {
log.Println(colorBlue("AccessKey: ") + colorBold(fmt.Sprintf("%s ", cred.AccessKey)))
log.Println(colorBlue("SecretKey: ") + colorBold(fmt.Sprintf("%s ", cred.SecretKey)))
log.Println(colorBlue("\nBrowser Access:"))
log.Println(fmt.Sprintf(getFormatStr(len(apiEndpointStr), 3), apiEndpointStr))
if globalIsBrowserEnabled {
log.Println(colorBlue("\nBrowser Access:"))
log.Println(fmt.Sprintf(getFormatStr(len(apiEndpointStr), 3), apiEndpointStr))
}
}
+22 -14
View File
@@ -18,10 +18,12 @@ package cmd
import (
"io"
"time"
"github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
"github.com/minio/minio/pkg/madmin"
)
// GatewayUnsupported list of unsupported call stubs for gateway.
@@ -37,8 +39,8 @@ func (a GatewayUnsupported) NewMultipartUpload(bucket string, object string, met
return "", errors.Trace(NotImplemented{})
}
// CopyObjectPart copy part of object to other bucket and object
func (a GatewayUnsupported) CopyObjectPart(srcBucket string, srcObject string, destBucket string, destObject string, uploadID string, partID int, startOffset int64, length int64, metadata map[string]string) (pi PartInfo, err error) {
// CopyObjectPart copy part of object to uploadID for another object
func (a GatewayUnsupported) CopyObjectPart(srcBucket, srcObject, destBucket, destObject, uploadID string, partID int, startOffset, length int64, metadata map[string]string, srcETag string) (pi PartInfo, err error) {
return pi, errors.Trace(NotImplemented{})
}
@@ -78,8 +80,8 @@ func (a GatewayUnsupported) DeleteBucketPolicies(bucket string) error {
}
// HealBucket - Not implemented stub
func (a GatewayUnsupported) HealBucket(bucket string) error {
return errors.Trace(NotImplemented{})
func (a GatewayUnsupported) HealBucket(bucket string, dryRun bool) ([]madmin.HealResultItem, error) {
return nil, errors.Trace(NotImplemented{})
}
// ListBucketsHeal - Not implemented stub
@@ -88,8 +90,8 @@ func (a GatewayUnsupported) ListBucketsHeal() (buckets []BucketInfo, err error)
}
// HealObject - Not implemented stub
func (a GatewayUnsupported) HealObject(bucket, object string) (int, int, error) {
return 0, 0, errors.Trace(NotImplemented{})
func (a GatewayUnsupported) HealObject(bucket, object string, dryRun bool) (h madmin.HealResultItem, e error) {
return h, errors.Trace(NotImplemented{})
}
// ListObjectsV2 - Not implemented stub
@@ -102,12 +104,6 @@ func (a GatewayUnsupported) ListObjectsHeal(bucket, prefix, marker, delimiter st
return loi, errors.Trace(NotImplemented{})
}
// ListUploadsHeal - Not implemented stub
func (a GatewayUnsupported) ListUploadsHeal(bucket, prefix, marker, uploadIDMarker,
delimiter string, maxUploads int) (lmi ListMultipartsInfo, e error) {
return lmi, errors.Trace(NotImplemented{})
}
// AnonListObjects - List objects anonymously
func (a GatewayUnsupported) AnonListObjects(bucket string, prefix string, marker string, delimiter string,
maxKeys int) (loi ListObjectsInfo, err error) {
@@ -132,7 +128,7 @@ func (a GatewayUnsupported) AnonPutObject(bucket, object string, data *hash.Read
}
// AnonGetObject downloads object anonymously.
func (a GatewayUnsupported) AnonGetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) (err error) {
func (a GatewayUnsupported) AnonGetObject(bucket, object string, startOffset int64, length int64, writer io.Writer, etag string) (err error) {
return errors.Trace(NotImplemented{})
}
@@ -143,6 +139,18 @@ func (a GatewayUnsupported) AnonGetObjectInfo(bucket, object string) (objInfo Ob
// CopyObject copies a blob from source container to destination container.
func (a GatewayUnsupported) CopyObject(srcBucket string, srcObject string, destBucket string, destObject string,
metadata map[string]string) (objInfo ObjectInfo, err error) {
metadata map[string]string, srcEtag string) (objInfo ObjectInfo, err error) {
return objInfo, errors.Trace(NotImplemented{})
}
// Locking operations
// ListLocks lists namespace locks held in object layer
func (a GatewayUnsupported) ListLocks(bucket, prefix string, duration time.Duration) ([]VolumeLockInfo, error) {
return []VolumeLockInfo{}, errors.Trace(NotImplemented{})
}
// ClearLocks clears namespace locks held in object layer
func (a GatewayUnsupported) ClearLocks([]VolumeLockInfo) error {
return errors.Trace(NotImplemented{})
}
+1 -1
View File
@@ -145,7 +145,7 @@ func (a *azureObjects) AnonGetBucketInfo(bucket string) (bucketInfo minio.Bucket
// AnonGetObject - SendGET request without authentication.
// This is needed when clients send GET requests on objects that can be downloaded without auth.
func (a *azureObjects) AnonGetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) (err error) {
func (a *azureObjects) AnonGetObject(bucket, object string, startOffset int64, length int64, writer io.Writer, etag string) (err error) {
h := make(http.Header)
if length > 0 && startOffset > 0 {
h.Add("Range", fmt.Sprintf("bytes=%d-%d", startOffset, startOffset+length-1))
+7 -3
View File
@@ -19,7 +19,6 @@ package azure
import (
"bytes"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
@@ -38,6 +37,7 @@ import (
"github.com/minio/minio/pkg/auth"
"github.com/minio/minio/pkg/errors"
"github.com/minio/minio/pkg/hash"
sha256 "github.com/minio/sha256-simd"
minio "github.com/minio/minio/cmd"
)
@@ -130,6 +130,10 @@ func (g *Azure) NewGatewayLayer(creds auth.Credentials) (minio.GatewayLayer, err
}
}
if endpoint == fmt.Sprintf("%s.blob.%s", creds.AccessKey, storage.DefaultBaseURL) {
// If, by mistake, user provides endpoint as accountname.blob.core.windows.net
endpoint = storage.DefaultBaseURL
}
c, err := storage.NewClient(creds.AccessKey, creds.SecretKey, endpoint, globalAzureAPIVersion, secure)
if err != nil {
return &azureObjects{}, err
@@ -554,7 +558,7 @@ func (a *azureObjects) ListObjectsV2(bucket, prefix, continuationToken, delimite
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (a *azureObjects) GetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) error {
func (a *azureObjects) GetObject(bucket, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
// startOffset cannot be negative.
if startOffset < 0 {
return azureToObjectError(errors.Trace(minio.InvalidRange{}), bucket, object)
@@ -621,7 +625,7 @@ func (a *azureObjects) PutObject(bucket, object string, data *hash.Reader, metad
// CopyObject - Copies a blob from source container to destination container.
// Uses Azure equivalent CopyBlob API.
func (a *azureObjects) CopyObject(srcBucket, srcObject, destBucket, destObject string, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
func (a *azureObjects) CopyObject(srcBucket, srcObject, destBucket, destObject string, metadata map[string]string, srcEtag string) (objInfo minio.ObjectInfo, err error) {
srcBlobURL := a.client.GetContainerReference(srcBucket).GetBlobReference(srcObject).GetURL()
destBlob := a.client.GetContainerReference(destBucket).GetBlobReference(destObject)
azureMeta, props, err := s3MetaToAzureProperties(metadata)
+1 -1
View File
@@ -43,7 +43,7 @@ func mkRange(offset, size int64) string {
// AnonGetObject - performs a plain http GET request on a public resource,
// fails if the resource is not public.
func (l *b2Objects) AnonGetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer) error {
func (l *b2Objects) AnonGetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
uri := fmt.Sprintf("%s/file/%s/%s", l.b2Client.DownloadURI, bucket, object)
req, err := http.NewRequest("GET", uri, nil)
if err != nil {
+2 -2
View File
@@ -379,7 +379,7 @@ func (l *b2Objects) ListObjectsV2(bucket, prefix, continuationToken, delimiter s
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (l *b2Objects) GetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer) error {
func (l *b2Objects) GetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
bkt, err := l.Bucket(bucket)
if err != nil {
return err
@@ -523,7 +523,7 @@ func (l *b2Objects) PutObject(bucket string, object string, data *h2.Reader, met
// CopyObject copies a blob from source container to destination container.
func (l *b2Objects) CopyObject(srcBucket string, srcObject string, dstBucket string,
dstObject string, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
dstObject string, metadata map[string]string, srcEtag string) (objInfo minio.ObjectInfo, err error) {
return objInfo, errors.Trace(minio.NotImplemented{})
}
+1 -1
View File
@@ -33,7 +33,7 @@ func toGCSPublicURL(bucket, object string) string {
}
// AnonGetObject - Get object anonymously
func (l *gcsGateway) AnonGetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer) error {
func (l *gcsGateway) AnonGetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
req, err := http.NewRequest("GET", toGCSPublicURL(bucket, object), nil)
if err != nil {
return gcsToObjectError(errors.Trace(err), bucket, object)
+2 -2
View File
@@ -725,7 +725,7 @@ func (l *gcsGateway) ListObjectsV2(bucket, prefix, continuationToken, delimiter
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (l *gcsGateway) GetObject(bucket string, key string, startOffset int64, length int64, writer io.Writer) error {
func (l *gcsGateway) GetObject(bucket string, key string, startOffset int64, length int64, writer io.Writer, etag string) error {
// if we want to mimic S3 behavior exactly, we need to verify if bucket exists first,
// otherwise gcs will just return object not exist in case of non-existing bucket
if _, err := l.client.Bucket(bucket).Attrs(l.ctx); err != nil {
@@ -813,7 +813,7 @@ func (l *gcsGateway) PutObject(bucket string, key string, data *hash.Reader, met
// CopyObject - Copies a blob from source container to destination container.
func (l *gcsGateway) CopyObject(srcBucket string, srcObject string, destBucket string, destObject string,
metadata map[string]string) (minio.ObjectInfo, error) {
metadata map[string]string, srcEtag string) (minio.ObjectInfo, error) {
src := l.client.Bucket(srcBucket).Object(srcObject)
dst := l.client.Bucket(destBucket).Object(destObject)
+28 -17
View File
@@ -27,12 +27,10 @@ import (
"path"
"strings"
"github.com/hashicorp/errwrap"
triton "github.com/joyent/triton-go"
"github.com/joyent/triton-go/authentication"
tclient "github.com/joyent/triton-go/client"
terrors "github.com/joyent/triton-go/errors"
"github.com/joyent/triton-go/storage"
"github.com/minio/cli"
minio "github.com/minio/minio/cmd"
"github.com/minio/minio/pkg/auth"
@@ -68,6 +66,7 @@ ENVIRONMENT VARIABLES:
MINIO_ACCESS_KEY: The Manta account name.
MINIO_SECRET_KEY: A KeyID associated with the Manta account.
MANTA_KEY_MATERIAL: The path to the SSH Key associated with the Manta account if the MINIO_SECRET_KEY is not in SSH Agent.
MANTA_SUBUSER: The username of a user who has limited access to your account.
BROWSER:
MINIO_BROWSER: To disable web browser access, set this value to "off".
@@ -140,7 +139,14 @@ func (g *Manta) NewGatewayLayer(creds auth.Credentials) (minio.GatewayLayer, err
keyMaterial := os.Getenv("MANTA_KEY_MATERIAL")
if keyMaterial == "" {
signer, err = authentication.NewSSHAgentSigner(creds.SecretKey, creds.AccessKey)
input := authentication.SSHAgentSignerInput{
KeyID: creds.SecretKey,
AccountName: creds.AccessKey,
}
if userName, ok := os.LookupEnv("MANTA_SUBUSER"); ok {
input.Username = userName
}
signer, err = authentication.NewSSHAgentSigner(input)
if err != nil {
return nil, errors.Trace(err)
}
@@ -168,7 +174,16 @@ func (g *Manta) NewGatewayLayer(creds auth.Credentials) (minio.GatewayLayer, err
keyBytes = []byte(keyMaterial)
}
signer, err = authentication.NewPrivateKeySigner(creds.SecretKey, keyBytes, creds.AccessKey)
input := authentication.PrivateKeySignerInput{
KeyID: creds.SecretKey,
PrivateKeyMaterial: keyBytes,
AccountName: creds.AccessKey,
}
if userName, ok := os.LookupEnv("MANTA_SUBUSER"); ok {
input.Username = userName
}
signer, err = authentication.NewPrivateKeySigner(input)
if err != nil {
return nil, errors.Trace(err)
}
@@ -324,7 +339,7 @@ func (t *tritonObjects) ListObjects(bucket, prefix, marker, delimiter string, ma
}
objs, err = t.client.Dir().List(ctx, input)
if err != nil {
if tclient.IsResourceNotFoundError(err) {
if terrors.IsResourceNotFoundError(err) {
return result, nil
}
return result, errors.Trace(err)
@@ -401,7 +416,7 @@ func (t *tritonObjects) ListObjectsV2(bucket, prefix, continuationToken, delimit
}
objs, err = t.client.Dir().List(ctx, input)
if err != nil {
if tclient.IsResourceNotFoundError(err) {
if terrors.IsResourceNotFoundError(err) {
return result, nil
}
return result, errors.Trace(err)
@@ -452,7 +467,7 @@ func (t *tritonObjects) ListObjectsV2(bucket, prefix, continuationToken, delimit
// indicates the total length of the object.
//
// https://apidocs.joyent.com/manta/api.html#GetObject
func (t *tritonObjects) GetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) error {
func (t *tritonObjects) GetObject(bucket, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
// Start offset cannot be negative.
if startOffset < 0 {
return errors.Trace(fmt.Errorf("Unexpected error"))
@@ -490,14 +505,10 @@ func (t *tritonObjects) GetObjectInfo(bucket, object string) (objInfo minio.Obje
ObjectPath: path.Join(mantaRoot, bucket, object),
})
if err != nil {
errType := &tclient.MantaError{}
if errwrap.ContainsType(err, errType) {
mantaErr := errwrap.GetType(err, errType).(*tclient.MantaError)
if mantaErr.StatusCode == http.StatusNotFound {
return objInfo, minio.ObjectNotFound{
Bucket: bucket,
Object: object,
}
if terrors.IsStatusNotFoundCode(err) {
return objInfo, minio.ObjectNotFound{
Bucket: bucket,
Object: object,
}
}
@@ -552,7 +563,7 @@ func (t *tritonObjects) PutObject(bucket, object string, data *hash.Reader, meta
// Uses Manta Snaplinks API.
//
// https://apidocs.joyent.com/manta/api.html#PutSnapLink
func (t *tritonObjects) CopyObject(srcBucket, srcObject, destBucket, destObject string, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
func (t *tritonObjects) CopyObject(srcBucket, srcObject, destBucket, destObject string, metadata map[string]string, srcEtag string) (objInfo minio.ObjectInfo, err error) {
ctx := context.Background()
if err = t.client.SnapLinks().Put(ctx, &storage.PutSnapLinkInput{
SourcePath: path.Join(mantaRoot, srcBucket, srcObject),
+2 -2
View File
@@ -29,8 +29,8 @@ func (l *ossObjects) AnonPutObject(bucket, object string, data *hash.Reader, met
}
// AnonGetObject - Get object anonymously
func (l *ossObjects) AnonGetObject(bucket, key string, startOffset, length int64, writer io.Writer) error {
return ossGetObject(l.anonClient, bucket, key, startOffset, length, writer)
func (l *ossObjects) AnonGetObject(bucket, key string, startOffset, length int64, writer io.Writer, etag string) error {
return ossGetObject(l.anonClient, bucket, key, startOffset, length, writer, etag)
}
// AnonGetObjectInfo - Get object info anonymously
+5 -5
View File
@@ -506,7 +506,7 @@ func (l *ossObjects) ListObjectsV2(bucket, prefix, continuationToken, delimiter
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func ossGetObject(client *oss.Client, bucket, key string, startOffset, length int64, writer io.Writer) error {
func ossGetObject(client *oss.Client, bucket, key string, startOffset, length int64, writer io.Writer, etag string) error {
if length < 0 && length != -1 {
return ossToObjectError(errors.Trace(fmt.Errorf("Invalid argument")), bucket, key)
}
@@ -539,8 +539,8 @@ func ossGetObject(client *oss.Client, bucket, key string, startOffset, length in
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (l *ossObjects) GetObject(bucket, key string, startOffset, length int64, writer io.Writer) error {
return ossGetObject(l.Client, bucket, key, startOffset, length, writer)
func (l *ossObjects) GetObject(bucket, key string, startOffset, length int64, writer io.Writer, etag string) error {
return ossGetObject(l.Client, bucket, key, startOffset, length, writer, etag)
}
func translatePlainError(err error) error {
@@ -618,7 +618,7 @@ func (l *ossObjects) PutObject(bucket, object string, data *hash.Reader, metadat
}
// CopyObject copies an object from source bucket to a destination bucket.
func (l *ossObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
func (l *ossObjects) CopyObject(srcBucket, srcObject, dstBucket, dstObject string, metadata map[string]string, srcEtag string) (objInfo minio.ObjectInfo, err error) {
bkt, err := l.Client.Bucket(srcBucket)
if err != nil {
return objInfo, ossToObjectError(errors.Trace(err), srcBucket, srcObject)
@@ -804,7 +804,7 @@ func ossListObjectParts(client *oss.Client, bucket, object, uploadID string, par
// CopyObjectPart creates a part in a multipart upload by copying
// existing object or a part of it.
func (l *ossObjects) CopyObjectPart(srcBucket, srcObject, destBucket, destObject, uploadID string,
partID int, startOffset, length int64, metadata map[string]string) (p minio.PartInfo, err error) {
partID int, startOffset, length int64, metadata map[string]string, srcEtag string) (p minio.PartInfo, err error) {
bkt, err := l.Client.Bucket(destBucket)
if err != nil {
+1 -1
View File
@@ -37,7 +37,7 @@ func (l *s3Objects) AnonPutObject(bucket string, object string, data *hash.Reade
}
// AnonGetObject - Get object anonymously
func (l *s3Objects) AnonGetObject(bucket string, key string, startOffset int64, length int64, writer io.Writer) error {
func (l *s3Objects) AnonGetObject(bucket string, key string, startOffset int64, length int64, writer io.Writer, etag string) error {
opts := miniogo.GetObjectOptions{}
if err := opts.SetRange(startOffset, startOffset+length-1); err != nil {
return minio.ErrorRespToObjectError(errors.Trace(err), bucket, key)
+3 -3
View File
@@ -254,7 +254,7 @@ func (l *s3Objects) ListObjectsV2(bucket, prefix, continuationToken, delimiter s
//
// startOffset indicates the starting read location of the object.
// length indicates the total length of the object.
func (l *s3Objects) GetObject(bucket string, key string, startOffset int64, length int64, writer io.Writer) error {
func (l *s3Objects) GetObject(bucket string, key string, startOffset int64, length int64, writer io.Writer, etag string) error {
if length < 0 && length != -1 {
return minio.ErrorRespToObjectError(errors.Trace(minio.InvalidRange{}), bucket, key)
}
@@ -298,7 +298,7 @@ func (l *s3Objects) PutObject(bucket string, object string, data *hash.Reader, m
}
// CopyObject copies an object from source bucket to a destination bucket.
func (l *s3Objects) CopyObject(srcBucket string, srcObject string, dstBucket string, dstObject string, metadata map[string]string) (objInfo minio.ObjectInfo, err error) {
func (l *s3Objects) CopyObject(srcBucket string, srcObject string, dstBucket string, dstObject string, metadata map[string]string, srcEtag string) (objInfo minio.ObjectInfo, err error) {
// Set this header such that following CopyObject() always sets the right metadata on the destination.
// metadata input is already a trickled down value from interpreting x-amz-metadata-directive at
// handler layer. So what we have right now is supposed to be applied on the destination object anyways.
@@ -354,7 +354,7 @@ func (l *s3Objects) PutObjectPart(bucket string, object string, uploadID string,
// CopyObjectPart creates a part in a multipart upload by copying
// existing object or a part of it.
func (l *s3Objects) CopyObjectPart(srcBucket, srcObject, destBucket, destObject, uploadID string,
partID int, startOffset, length int64, metadata map[string]string) (p minio.PartInfo, err error) {
partID int, startOffset, length int64, metadata map[string]string, srcEtag string) (p minio.PartInfo, err error) {
completePart, err := l.Client.CopyObjectPart(srcBucket, srcObject, destBucket, destObject,
uploadID, partID, startOffset, length, metadata)
+1 -1
View File
@@ -410,7 +410,7 @@ func (s *siaObjects) ListObjects(bucket string, prefix string, marker string, de
return loi, nil
}
func (s *siaObjects) GetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer) error {
func (s *siaObjects) GetObject(bucket string, object string, startOffset int64, length int64, writer io.Writer, etag string) error {
dstFile := path.Join(s.TempDir, minio.MustGetUUID())
defer os.Remove(dstFile)
+24 -19
View File
@@ -229,7 +229,7 @@ func setBrowserCacheControlHandler(h http.Handler) http.Handler {
}
func (h cacheControlHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method == httpGET && guessIsBrowserReq(r) && globalIsBrowserEnabled {
if r.Method == http.MethodGet && guessIsBrowserReq(r) && globalIsBrowserEnabled {
// For all browser requests set appropriate Cache-Control policies
if hasPrefix(r.URL.Path, minioReservedBucketPath+"/") {
if hasSuffix(r.URL.Path, ".js") || r.URL.Path == minioReservedBucketPath+"/favicon.ico" {
@@ -246,6 +246,12 @@ func (h cacheControlHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
h.handler.ServeHTTP(w, r)
}
// Check to allow access to the reserved "bucket" `/minio` for Admin
// API requests.
func isAdminReq(r *http.Request) bool {
return strings.HasPrefix(r.URL.Path, adminAPIPathPrefix+"/")
}
// Adds verification for incoming paths.
type minioReservedBucketHandler struct {
handler http.Handler
@@ -256,8 +262,12 @@ func setReservedBucketHandler(h http.Handler) http.Handler {
}
func (h minioReservedBucketHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if !guessIsRPCReq(r) && !guessIsBrowserReq(r) {
// For all non browser, non RPC requests, reject access to 'minioReservedBucketPath'.
switch {
case guessIsRPCReq(r), guessIsBrowserReq(r), isAdminReq(r):
// Allow access to reserved buckets
default:
// For all other requests reject access to reserved
// buckets
bucketName, _ := urlPath2BucketObjectName(r.URL)
if isMinioReservedBucket(bucketName) || isMinioMetaBucket(bucketName) {
writeErrorResponse(w, ErrAllAccessDisabled, r.URL)
@@ -341,24 +351,14 @@ type resourceHandler struct {
handler http.Handler
}
// List of http methods.
const (
httpGET = "GET"
httpPUT = "PUT"
httpHEAD = "HEAD"
httpPOST = "POST"
httpDELETE = "DELETE"
httpOPTIONS = "OPTIONS"
)
// List of default allowable HTTP methods.
var defaultAllowableHTTPMethods = []string{
httpGET,
httpPUT,
httpHEAD,
httpPOST,
httpDELETE,
httpOPTIONS,
http.MethodGet,
http.MethodPut,
http.MethodHead,
http.MethodPost,
http.MethodDelete,
http.MethodOptions,
}
// setCorsHandler handler for CORS (Cross Origin Resource Sharing)
@@ -443,6 +443,11 @@ func (h resourceHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return
}
}
// A put method on path "/" doesn't make sense, ignore it.
if r.Method == http.MethodPut && r.URL.Path == "/" {
writeErrorResponse(w, ErrNotImplemented, r.URL)
return
}
// Serve HTTP.
h.handler.ServeHTTP(w, r)
+8 -5
View File
@@ -123,8 +123,8 @@ var (
globalHTTPServerErrorCh = make(chan error)
globalOSSignalCh = make(chan os.Signal, 1)
// Enable HTTP request/response headers and body logging.
globalHTTPTrace bool
// File to log HTTP request/response headers and body.
globalHTTPTraceFile *os.File
// List of admin peers.
globalAdminPeers = adminPeers{}
@@ -143,9 +143,8 @@ var (
// Time when object layer was initialized on start up.
globalBootTime time.Time
globalActiveCred auth.Credentials
globalPublicCerts []*x509.Certificate
globalXLObjCacheDisabled bool
globalActiveCred auth.Credentials
globalPublicCerts []*x509.Certificate
globalIsEnvDomainName bool
globalDomainName string // Root domain for virtual host style requests
@@ -166,6 +165,9 @@ var (
// Set to store standard storage class
globalStandardStorageClass storageClass
// RPC version.
globalRPCAPIVersion = semVersion{1, 0, 0}
// Add new variable global values here.
)
@@ -174,6 +176,7 @@ var (
colorBold = color.New(color.Bold).SprintFunc()
colorBlue = color.New(color.FgBlue).SprintfFunc()
colorYellow = color.New(color.FgYellow).SprintfFunc()
colorRed = color.New(color.FgRed).SprintfFunc()
)
// Returns minio global information, as a key value map.
+4 -5
View File
@@ -22,7 +22,6 @@ import (
"net"
"net/http"
"net/url"
"os"
"strings"
"github.com/minio/minio/pkg/errors"
@@ -251,18 +250,18 @@ func extractPostPolicyFormValues(form *multipart.Form) (filePart io.ReadCloser,
// Log headers and body.
func httpTraceAll(f http.HandlerFunc) http.HandlerFunc {
if !globalHTTPTrace {
if globalHTTPTraceFile == nil {
return f
}
return httptracer.TraceReqHandlerFunc(f, os.Stdout, true)
return httptracer.TraceReqHandlerFunc(f, globalHTTPTraceFile, true)
}
// Log only the headers.
func httpTraceHdrs(f http.HandlerFunc) http.HandlerFunc {
if !globalHTTPTrace {
if globalHTTPTraceFile == nil {
return f
}
return httptracer.TraceReqHandlerFunc(f, os.Stdout, false)
return httptracer.TraceReqHandlerFunc(f, globalHTTPTraceFile, false)
}
// Returns "/bucketName/objectName" for path-style or virtual-host-style requests.
+4 -7
View File
@@ -63,14 +63,11 @@ func authenticateJWT(accessKey, secretKey string, expiry time.Duration) (string,
return "", errAuthentication
}
utcNow := UTCNow()
token := jwtgo.NewWithClaims(jwtgo.SigningMethodHS512, jwtgo.StandardClaims{
ExpiresAt: utcNow.Add(expiry).Unix(),
IssuedAt: utcNow.Unix(),
jwt := jwtgo.NewWithClaims(jwtgo.SigningMethodHS512, jwtgo.StandardClaims{
ExpiresAt: UTCNow().Add(expiry).Unix(),
Subject: accessKey,
})
return token.SignedString([]byte(serverCred.SecretKey))
return jwt.SignedString([]byte(serverCred.SecretKey))
}
func authenticateNode(accessKey, secretKey string) (string, error) {
@@ -127,7 +124,7 @@ func webRequestAuthenticate(req *http.Request) error {
return errAuthentication
}
if err = claims.Valid(); err != nil {
return err
return errAuthentication
}
if claims.Subject != globalServerConfig.GetCredential().AccessKey {
return errInvalidAccessKeyID
+2 -2
View File
@@ -39,8 +39,8 @@ func testAuthenticate(authType string, t *testing.T) {
secretKey string
expectedErr error
}{
// Access key (less than 5 chrs) too small.
{"user", cred.SecretKey, auth.ErrInvalidAccessKeyLength},
// Access key (less than 3 chrs) too small.
{"u1", cred.SecretKey, auth.ErrInvalidAccessKeyLength},
// Secret key (less than 8 chrs) too small.
{cred.AccessKey, "pass", auth.ErrInvalidSecretKeyLength},
// Authentication error.
-2
View File
@@ -201,8 +201,6 @@ func (n *nsLockMap) statusBlockedToNone(param nsParam, lockSource, opsID string,
if lockInfo.status != blockedStatus {
return errors.Trace(LockInfoStateNotBlocked{param.volume, param.path, opsID})
}
// Clear the status by removing the entry for the given `opsID`.
delete(n.debugLockMap[param].lockInfo, opsID)
// Update global lock stats.
n.counters.lockTimedOut()
+30
View File
@@ -637,3 +637,33 @@ func TestNsLockMapDeleteLockInfoEntryForVolumePath(t *testing.T) {
t.Errorf("Expected the count of all locks to be %v, but got %v", 0, globalNSMutex.counters.total)
}
}
// Test to assert that status change from blocked to none shouldn't remove lock info entry for ops
// Ref: Logs from https://github.com/minio/minio/issues/5311
func TestStatusBlockedToNone(t *testing.T) {
// Initialize namespace lock subsystem
initNSLock(false)
ns := globalNSMutex
volume, path := "bucket", "object"
param := nsParam{volume: volume, path: path}
lockSrc := "main.go:1"
opsID := "1"
err := ns.statusNoneToBlocked(param, lockSrc, opsID, false)
if err != nil {
t.Fatal("Failed to mark lock state to blocked")
}
err = ns.statusBlockedToNone(param, lockSrc, opsID, false)
if err != nil {
t.Fatal("Failed to mark lock state to none")
}
err = ns.deleteLockInfoEntryForOps(param, opsID)
if err != nil {
t.Fatalf("Expected deleting of lock entry for %s to pass but got %v", opsID, err)
}
}
+26 -30
View File
@@ -19,7 +19,6 @@ package cmd
import (
"fmt"
"math/rand"
"path"
"sync"
"time"
@@ -64,48 +63,45 @@ type lockServer struct {
}
// Start lock maintenance from all lock servers.
func startLockMaintenance(lockServers []*lockServer) {
for _, locker := range lockServers {
// Start loop for stale lock maintenance
go func(lk *lockServer) {
// Initialize a new ticker with a minute between each ticks.
ticker := time.NewTicker(lockMaintenanceInterval)
func startLockMaintenance(lkSrv *lockServer) {
// Start loop for stale lock maintenance
go func(lk *lockServer) {
// Initialize a new ticker with a minute between each ticks.
ticker := time.NewTicker(lockMaintenanceInterval)
// Start with random sleep time, so as to avoid "synchronous checks" between servers
time.Sleep(time.Duration(rand.Float64() * float64(lockMaintenanceInterval)))
for {
// Verifies every minute for locks held more than 2minutes.
select {
case <-ticker.C:
lk.lockMaintenance(lockValidityCheckInterval)
case <-globalServiceDoneCh:
// Stop the timer.
ticker.Stop()
}
// Start with random sleep time, so as to avoid "synchronous checks" between servers
time.Sleep(time.Duration(rand.Float64() * float64(lockMaintenanceInterval)))
for {
// Verifies every minute for locks held more than 2minutes.
select {
case <-globalServiceDoneCh:
// Stop the timer upon service closure and cleanup the go-routine.
ticker.Stop()
return
case <-ticker.C:
lk.lockMaintenance(lockValidityCheckInterval)
}
}(locker)
}
}
}(lkSrv)
}
// Register distributed NS lock handlers.
func registerDistNSLockRouter(mux *router.Router, endpoints EndpointList) error {
// Start lock maintenance from all lock servers.
startLockMaintenance(globalLockServers)
startLockMaintenance(globalLockServer)
// Register initialized lock servers to their respective rpc endpoints.
return registerStorageLockers(mux, globalLockServers)
return registerStorageLockers(mux, globalLockServer)
}
// registerStorageLockers - register locker rpc handlers for net/rpc library clients
func registerStorageLockers(mux *router.Router, lockServers []*lockServer) error {
for _, lockServer := range lockServers {
lockRPCServer := newRPCServer()
if err := lockRPCServer.RegisterName(lockServiceName, lockServer); err != nil {
return errors.Trace(err)
}
lockRouter := mux.PathPrefix(minioReservedBucketPath).Subrouter()
lockRouter.Path(path.Join(lockServicePath, lockServer.ll.serviceEndpoint)).Handler(lockRPCServer)
func registerStorageLockers(mux *router.Router, lkSrv *lockServer) error {
lockRPCServer := newRPCServer()
if err := lockRPCServer.RegisterName(lockServiceName, lkSrv); err != nil {
return errors.Trace(err)
}
lockRouter := mux.PathPrefix(minioReservedBucketPath).Subrouter()
lockRouter.Path(lockServicePath).Handler(lockRPCServer)
return nil
}
+25 -12
View File
@@ -64,7 +64,7 @@ func createLockTestServer(t *testing.T) (string, *lockServer, string) {
}
loginArgs := LoginRPCArgs{
AuthToken: token,
Version: Version,
Version: globalRPCAPIVersion,
RequestTime: UTCNow(),
}
loginReply := LoginRPCReply{}
@@ -87,6 +87,7 @@ func TestLockRpcServerLock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la.SetAuthToken(token)
la.SetRPCAPIVersion(globalRPCAPIVersion)
// Claim a lock
var result bool
@@ -120,6 +121,7 @@ func TestLockRpcServerLock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la2.SetAuthToken(token)
la2.SetRPCAPIVersion(globalRPCAPIVersion)
err = locker.Lock(&la2, &result)
if err != nil {
@@ -143,6 +145,7 @@ func TestLockRpcServerUnlock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la.SetAuthToken(token)
la.SetRPCAPIVersion(globalRPCAPIVersion)
// First test return of error when attempting to unlock a lock that does not exist
var result bool
@@ -188,6 +191,7 @@ func TestLockRpcServerRLock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la.SetAuthToken(token)
la.SetRPCAPIVersion(globalRPCAPIVersion)
// Claim a lock
var result bool
@@ -221,6 +225,7 @@ func TestLockRpcServerRLock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la2.SetAuthToken(token)
la2.SetRPCAPIVersion(globalRPCAPIVersion)
err = locker.RLock(&la2, &result)
if err != nil {
@@ -244,6 +249,7 @@ func TestLockRpcServerRUnlock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la.SetAuthToken(token)
la.SetRPCAPIVersion(globalRPCAPIVersion)
// First test return of error when attempting to unlock a read-lock that does not exist
var result bool
@@ -268,6 +274,7 @@ func TestLockRpcServerRUnlock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la2.SetAuthToken(token)
la2.SetRPCAPIVersion(globalRPCAPIVersion)
// ... and create a second lock on same resource
err = locker.RLock(&la2, &result)
@@ -330,6 +337,7 @@ func TestLockRpcServerForceUnlock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
laForce.SetAuthToken(token)
laForce.SetRPCAPIVersion(globalRPCAPIVersion)
// First test that UID should be empty
var result bool
@@ -352,6 +360,7 @@ func TestLockRpcServerForceUnlock(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la.SetAuthToken(token)
la.SetRPCAPIVersion(globalRPCAPIVersion)
// Create lock ... (so that we can force unlock)
err = locker.Lock(&la, &result)
@@ -394,6 +403,7 @@ func TestLockRpcServerExpired(t *testing.T) {
ServiceEndpoint: "rpc-path",
})
la.SetAuthToken(token)
la.SetRPCAPIVersion(globalRPCAPIVersion)
// Unknown lock at server will return expired = true
var expired bool
@@ -425,8 +435,8 @@ func TestLockRpcServerExpired(t *testing.T) {
}
}
// Test initialization of lock servers.
func TestLockServers(t *testing.T) {
// Test initialization of lock server.
func TestLockServerInit(t *testing.T) {
if runtime.GOOS == globalWindowsOSName {
return
}
@@ -438,8 +448,10 @@ func TestLockServers(t *testing.T) {
defer os.RemoveAll(rootPath)
currentIsDistXL := globalIsDistXL
currentLockServer := globalLockServer
defer func() {
globalIsDistXL = currentIsDistXL
globalLockServer = currentLockServer
}()
case1Endpoints := mustGetNewEndpointList(
@@ -468,26 +480,27 @@ func TestLockServers(t *testing.T) {
globalMinioHost = ""
testCases := []struct {
isDistXL bool
endpoints EndpointList
totalLockServers int
isDistXL bool
endpoints EndpointList
}{
// Test - 1 one lock server initialized.
{true, case1Endpoints, 1},
// Test - 2 two servers possible.
{true, case2Endpoints, 2},
{true, case1Endpoints},
// Test - similar endpoint hosts should
// converge to single lock server
// initialized.
{true, case2Endpoints},
}
// Validates lock server initialization.
for i, testCase := range testCases {
globalIsDistXL = testCase.isDistXL
globalLockServers = nil
globalLockServer = nil
_, _ = newDsyncNodes(testCase.endpoints)
if err != nil {
t.Fatalf("Got unexpected error initializing lock servers: %v", err)
}
if len(globalLockServers) != testCase.totalLockServers {
t.Fatalf("Test %d: Expected total %d, got %d", i+1, testCase.totalLockServers, len(globalLockServers))
if globalLockServer == nil && testCase.isDistXL {
t.Errorf("Test %d: Expected initialized lockServer, but got uninitialized", i+1)
}
}
}
+2 -43
View File
@@ -66,47 +66,6 @@ type OpsLockState struct {
// listLocksInfo - Fetches locks held on bucket, matching prefix held for longer than duration.
func listLocksInfo(bucket, prefix string, duration time.Duration) []VolumeLockInfo {
globalNSMutex.lockMapMutex.Lock()
defer globalNSMutex.lockMapMutex.Unlock()
// Fetch current time once instead of fetching system time for every lock.
timeNow := UTCNow()
volumeLocks := []VolumeLockInfo{}
for param, debugLock := range globalNSMutex.debugLockMap {
if param.volume != bucket {
continue
}
// N B empty prefix matches all param.path.
if !hasPrefix(param.path, prefix) {
continue
}
volLockInfo := VolumeLockInfo{
Bucket: param.volume,
Object: param.path,
LocksOnObject: debugLock.counters.total,
TotalBlockedLocks: debugLock.counters.blocked,
LocksAcquiredOnObject: debugLock.counters.granted,
}
// Filter locks that are held on bucket, prefix.
for opsID, lockInfo := range debugLock.lockInfo {
// filter locks that were held for longer than duration.
elapsed := timeNow.Sub(lockInfo.since)
if elapsed < duration {
continue
}
// Add locks that are held for longer than duration.
volLockInfo.LockDetailsOnObject = append(volLockInfo.LockDetailsOnObject,
OpsLockState{
OperationID: opsID,
LockSource: lockInfo.lockSource,
LockType: lockInfo.lType,
Status: lockInfo.status,
Since: lockInfo.since,
})
volumeLocks = append(volumeLocks, volLockInfo)
}
}
return volumeLocks
locksInfo, _ := newObjectLayerFn().ListLocks(bucket, prefix, duration)
return locksInfo
}
+26 -4
View File
@@ -18,21 +18,43 @@ package cmd
import (
"fmt"
"os"
"testing"
"time"
)
// TestListLocksInfo - Test for listLocksInfo.
func TestListLocksInfo(t *testing.T) {
// Initialize globalNSMutex to validate listing of lock
// instrumentation information.
// reset global variables to start afresh.
resetTestGlobals()
// Initialize minio server config.
rootPath, err := newTestConfig(globalMinioDefaultRegion)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(rootPath)
// Initializing new XL objectLayer.
objAPI, _, xlErr := initTestXLObjLayer()
if xlErr != nil {
t.Fatalf("failed to init object layer")
}
// Make objLayer available to all internal services via globalObjectAPI.
globalObjLayerMutex.Lock()
globalObjectAPI = objAPI
globalObjLayerMutex.Unlock()
// Set globalIsXL to indicate that the setup uses an erasure code backend.
// initialize NSLock.
isDistXL := false
initNSLock(isDistXL)
var nsMutex *nsLockMap
nsMutex = objAPI.(*xlObjects).nsMutex
// Acquire a few locks to populate lock instrumentation.
// Take 10 read locks on bucket1/prefix1/obj1
for i := 0; i < 10; i++ {
readLk := globalNSMutex.NewNSLock("bucket1", "prefix1/obj1")
readLk := nsMutex.NewNSLock("bucket1", "prefix1/obj1")
if readLk.GetRLock(newDynamicTimeout(60*time.Second, time.Second)) != nil {
t.Errorf("Failed to get read lock on iteration %d", i)
}
@@ -40,7 +62,7 @@ func TestListLocksInfo(t *testing.T) {
// Take write locks on bucket1/prefix/obj{11..19}
for i := 0; i < 10; i++ {
wrLk := globalNSMutex.NewNSLock("bucket1", fmt.Sprintf("prefix1/obj%d", 10+i))
wrLk := nsMutex.NewNSLock("bucket1", fmt.Sprintf("prefix1/obj%d", 10+i))
if wrLk.GetLock(newDynamicTimeout(60*time.Second, time.Second)) != nil {
t.Errorf("Failed to get write lock on iteration %d", i)
}
+135 -101
View File
@@ -17,81 +17,71 @@
package cmd
import (
"encoding/json"
"fmt"
"io/ioutil"
"path"
"os"
"path/filepath"
"runtime"
"strings"
"time"
"github.com/Sirupsen/logrus"
"github.com/minio/mc/pkg/console"
"github.com/minio/minio/pkg/errors"
)
var log = NewLogger()
var trimStrings []string
// LogTarget - interface for log target.
type LogTarget interface {
Fire(entry *logrus.Entry) error
String() string
// Level type
type Level int8
// Enumerated level types
const (
Error Level = iota + 1
Fatal
)
func (level Level) String() string {
var lvlStr string
switch level {
case Error:
lvlStr = "ERROR"
case Fatal:
lvlStr = "FATAL"
}
return lvlStr
}
// BaseLogTarget - base log target.
type BaseLogTarget struct {
Enable bool `json:"enable"`
formatter logrus.Formatter
type logEntry struct {
Level string `json:"level"`
Message string `json:"message"`
Time string `json:"time"`
Cause string `json:"cause"`
Trace []string `json:"trace"`
}
// Logger - higher level logger.
// Logger - for console messages
type Logger struct {
logger *logrus.Logger
consoleTarget ConsoleLogger
targets []LogTarget
quiet bool
quiet bool
json bool
}
// AddTarget - add logger to this hook.
func (log *Logger) AddTarget(logTarget LogTarget) {
log.targets = append(log.targets, logTarget)
// NewLogger - to create a new Logger object
func NewLogger() *Logger {
return &Logger{}
}
// SetConsoleTarget - sets console target to this hook.
func (log *Logger) SetConsoleTarget(consoleTarget ConsoleLogger) {
log.consoleTarget = consoleTarget
}
// Fire - log entry handler to save logs.
func (log *Logger) Fire(entry *logrus.Entry) (err error) {
if err = log.consoleTarget.Fire(entry); err != nil {
log.Printf("Unable to log to console target. %s\n", err)
}
for _, logTarget := range log.targets {
if err = logTarget.Fire(entry); err != nil {
log.Printf("Unable to log to target %s. %s\n", logTarget, err)
}
}
return err
}
// Levels - returns list of log levels support.
func (log *Logger) Levels() []logrus.Level {
return []logrus.Level{
logrus.PanicLevel,
logrus.FatalLevel,
logrus.ErrorLevel,
logrus.WarnLevel,
logrus.InfoLevel,
logrus.DebugLevel,
}
}
// EnableQuiet - sets quiet option.
// EnableQuiet - turns quiet option on.
func (log *Logger) EnableQuiet() {
log.quiet = true
}
// EnableJSON - outputs logs in json format.
func (log *Logger) EnableJSON() {
log.json = true
log.quiet = true
}
// Println - wrapper to console.Println() with quiet flag.
func (log *Logger) Println(args ...interface{}) {
if !log.quiet {
@@ -106,38 +96,66 @@ func (log *Logger) Printf(format string, args ...interface{}) {
}
}
// NewLogger - returns a new initialized logger.
func NewLogger() *Logger {
logger := logrus.New()
logger.Out = ioutil.Discard
logger.Level = logrus.DebugLevel
l := &Logger{
logger: logger,
consoleTarget: NewConsoleLogger(),
}
// Adds a console logger.
logger.Hooks.Add(l)
return l
}
func getSource() string {
var funcName string
pc, filename, lineNum, ok := runtime.Caller(2)
if ok {
filename = path.Base(filename)
funcName = strings.TrimPrefix(runtime.FuncForPC(pc).Name(), "github.com/minio/minio/cmd.")
func init() {
var goPathList []string
// Add all possible GOPATH paths into trimStrings
// Split GOPATH depending on the OS type
if runtime.GOOS == "windows" {
goPathList = strings.Split(GOPATH, ";")
} else {
filename = "<unknown>"
lineNum = 0
// All other types of OSs
goPathList = strings.Split(GOPATH, ":")
}
return fmt.Sprintf("[%s:%d:%s()]", filename, lineNum, funcName)
// Add trim string "{GOROOT}/src/" into trimStrings
trimStrings = []string{filepath.Join(runtime.GOROOT(), "src") + string(filepath.Separator)}
// Add all possible path from GOPATH=path1:path2...:pathN
// as "{path#}/src/" into trimStrings
for _, goPathString := range goPathList {
trimStrings = append(trimStrings, filepath.Join(goPathString, "src")+string(filepath.Separator))
}
// Add "github.com/minio/minio" as the last to cover
// paths like "{GOROOT}/src/github.com/minio/minio"
// and "{GOPATH}/src/github.com/minio/minio"
trimStrings = append(trimStrings, filepath.Join("github.com", "minio", "minio")+string(filepath.Separator))
}
func logIf(level logrus.Level, source string, err error, msg string, data ...interface{}) {
func trimTrace(f string) string {
for _, trimString := range trimStrings {
f = strings.TrimPrefix(filepath.ToSlash(f), filepath.ToSlash(trimString))
}
return filepath.FromSlash(f)
}
// getTrace method - creates and returns stack trace
func getTrace(traceLevel int) []string {
var trace []string
pc, file, lineNumber, ok := runtime.Caller(traceLevel)
for ok {
// Clean up the common prefixes
file = trimTrace(file)
// Get the function name
_, funcName := filepath.Split(runtime.FuncForPC(pc).Name())
// Skip duplicate traces that start with file name, "<autogenerated>"
// and also skip traces with function name that starts with "runtime."
if !strings.HasPrefix(file, "<autogenerated>") &&
!strings.HasPrefix(funcName, "runtime.") {
// Form and append a line of stack trace into a
// collection, 'trace', to build full stack trace
trace = append(trace, fmt.Sprintf("%v:%v:%v()", file, lineNumber, funcName))
}
traceLevel++
// Read stack trace information from PC
pc, file, lineNumber, ok = runtime.Caller(traceLevel)
}
return trace
}
func logIf(level Level, err error, msg string,
data ...interface{}) {
isErrIgnored := func(err error) (ok bool) {
err = errors.Cause(err)
switch err.(type) {
@@ -154,36 +172,52 @@ func logIf(level logrus.Level, source string, err error, msg string, data ...int
if err == nil || isErrIgnored(err) {
return
}
// Get the cause for the Error
cause := err.Error()
// Get full stack trace
trace := getTrace(3)
// Get time
timeOfError := UTCNow().Format(time.RFC3339Nano)
// Output the formatted log message at console
var output string
message := fmt.Sprintf(msg, data...)
if log.json {
logJSON, err := json.Marshal(&logEntry{
Level: level.String(),
Message: message,
Time: timeOfError,
Cause: cause,
Trace: trace,
})
if err != nil {
panic("json marshal of logEntry failed: " + err.Error())
}
output = string(logJSON)
} else {
// Add a sequence number and formatting for each stack trace
// No formatting is required for the first entry
trace[0] = "1: " + trace[0]
for i, element := range trace[1:] {
trace[i+1] = fmt.Sprintf("%8v: %s", i+2, element)
}
errMsg := fmt.Sprintf("[%s] [%s] %s (%s)",
timeOfError, level.String(), message, cause)
fields := logrus.Fields{
"source": source,
"cause": err.Error(),
output = fmt.Sprintf("\nTrace: %s\n%s",
strings.Join(trace, "\n"),
colorRed(colorBold(errMsg)))
}
fmt.Println(output)
if terr, ok := err.(*errors.Error); ok {
fields["stack"] = strings.Join(terr.Stack(), " ")
}
switch level {
case logrus.PanicLevel:
log.logger.WithFields(fields).Panicf(msg, data...)
case logrus.FatalLevel:
log.logger.WithFields(fields).Fatalf(msg, data...)
case logrus.ErrorLevel:
log.logger.WithFields(fields).Errorf(msg, data...)
case logrus.WarnLevel:
log.logger.WithFields(fields).Warnf(msg, data...)
case logrus.InfoLevel:
log.logger.WithFields(fields).Infof(msg, data...)
default:
log.logger.WithFields(fields).Debugf(msg, data...)
if level == Fatal {
os.Exit(1)
}
}
func errorIf(err error, msg string, data ...interface{}) {
logIf(logrus.ErrorLevel, getSource(), err, msg, data...)
logIf(Error, err, msg, data...)
}
func fatalIf(err error, msg string, data ...interface{}) {
logIf(logrus.FatalLevel, getSource(), err, msg, data...)
logIf(Fatal, err, msg, data...)
}

Some files were not shown because too many files have changed in this diff Show More