Add erasure code.

XL: Implement ListObjects channel and pool management.

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,13 @@
+## Expected behaviour
+
+## Actual behaviour
+
+## Steps to reproduce the behaviour
+
+## Minio version
+- (paste output of `minio version`)
+
+## System information
+
+
+

.gitignore

@@ -13,3 +13,5 @@ build
 vendor/**/*.js
 vendor/**/*.json
 release
+.DS_Store
+*.syso

.mention-bot

@@ -0,0 +1,4 @@
+{
+    "numFilesToCheck": 10,
+    "requiredOrgs": ["minio"]
+}

CONTRIBUTING.md

@@ -3,11 +3,11 @@
 If you do not have a working Golang environment setup please follow [Golang Installation Guide](./INSTALLGO.md).
 
 ### Setup your Minio Github Repository
-Fork [Minio upstream](https://github.com/minio/minio/fork) source repository to your own personal repository. Copy the URL and pass it to ``go get`` command. Go uses git to clone a copy into your project workspace folder.
+Fork [Minio upstream](https://github.com/minio/minio/fork) source repository to your own personal repository. Copy the URL for minio from your personal github repo (you will need it for the `git clone` command below).
 ```sh
 $ mkdir -p $GOPATH/src/github.com/minio
 $ cd $GOPATH/src/github.com/minio
-$ git clone https://github.com/$USER_ID/minio
+$ git clone <paste saved URL for personal forked minio repo>
 $ cd minio
 ```
 

Dockerfile

@@ -0,0 +1,15 @@
+FROM golang:1.6
+
+RUN mkdir -p /go/src/app
+WORKDIR /go/src/app
+
+COPY . /go/src/app
+RUN go-wrapper download
+RUN go-wrapper install
+
+ENV ALLOW_CONTAINER_ROOT=1
+RUN mkdir -p /export/docker && cp /go/src/app/Docker.md /export/docker/
+
+EXPOSE 9000
+ENTRYPOINT ["go-wrapper", "run", "server"]
+CMD ["/export"]

FreeBSD.md

@@ -0,0 +1,156 @@
+### Minio with ZFS backend - FreeBSD
+
+This example assumes that you have a FreeBSD 10.x running
+
+#### Step *1* 
+
+As root on the FreeBSD edit `/etc/rc.conf`
+```
+zfs_enable="YES"
+```
+
+Start ZFS service
+```
+# service zfs start
+```
+
+```
+# dd if=/dev/zero of=/zfs bs=1M count=4000
+```
+
+Configure a loopback device on the `/zfs` file. 
+```
+# mdconfig -a -t vnode -f /zfs
+```
+
+Create zfs pool
+```
+# zpool create minio-example /dev/md0
+```
+
+```
+# df /minio-example
+Filesystem    512-blocks Used   Avail Capacity  Mounted on
+minio-example    7872440   38 7872402     0%    /minio-example
+```
+
+Verify if it is writable
+```
+# touch /minio-example/testfile
+# ls -l /minio-example/testfile
+-rw-r--r--  1 root  wheel  0 Apr 26 00:51 /minio-example/testfile
+```
+
+Now you have successfully created a ZFS pool for futher reading please refer to [ZFS Quickstart Guide](https://www.freebsd.org/doc/handbook/zfs-quickstart.html)
+
+However, this pool is not taking advantage of any ZFS features, so let's create a ZFS filesytem on this pool with compression enabled.  ZFS supports many compression algorithms: lzjb, gzip, zle, lz4.  LZ4 is often the most performant algorithm in terms of compression of data versus system overhead.
+
+
+```
+# zfs create minio-example/compressed-objects
+# zfs set compression=lz4 minio-example/compressed-objects
+```
+
+To keep monitoring your pool use
+
+```
+# zpool status
+  pool: minio-example
+ state: ONLINE
+  scan: none requested
+config:
+
+	NAME        STATE     READ WRITE CKSUM
+	minio-example  ONLINE       0     0     0
+	  md0       ONLINE       0     0     0
+
+errors: No known data errors
+```
+
+#### Step *2*
+
+Now start minio server on the ``/minio-example/compressed-objects``, change the permissions such that this directory is accessibly by a normal user
+
+```
+# chown -R minio-user:minio-user /minio-example/compressed-objects
+```
+
+Now login as ``minio-user`` and start minio server. 
+
+```
+$ curl https://dl.minio.io/server/minio/release/freebsd-amd64/minio > minio
+$ chmod 755 minio
+$ ./minio server /minio-example/compressed-objects
+
+AccessKey: X3RU3Q6B4T20TYB281W5  SecretKey: bvp0DF8c+6MzL60UxFqOZGlA5Z8UPfIAATUQuzLS  Region: us-east-1
+
+Minio Object Storage:
+    http://159.203.234.49:9000
+    http://10.12.0.5:9000
+    http://127.0.0.1:9000
+
+Minio Browser:
+    http://159.203.234.49:9000
+    http://10.12.0.5:9000
+    http://127.0.0.1:9000
+
+To configure Minio Client:
+    $ wget https://dl.minio.io/client/mc/release/freebsd-amd64/mc
+    $ chmod 755 mc
+    $ ./mc config host add myminio http://localhost:9000 X3RU3Q6B4T20TYB281W5 bvp0DF8c+6MzL60UxFqOZGlA5Z8UPfIAATUQuzLS
+```
+
+Point your browser to http://localhost:9000 and login with the credentials displayed on the command line. 
+
+Now you have a S3 compatible server running on top of your ZFS backend which transparently provides disk level compression for your uploaded objects.
+
+Thanks for using Minio, awaiting feedback :-) 
+
+#### Building Minio Server From Source
+
+It is possible to build the minio server from source on FreeBSD.  To do this we will used the golang distribution provided by the FreeBSD pkg infrastructure.
+
+First we will need to install golang as well as GNU make:
+
+```
+$ sudo pkg install go gmake
+Updating FreeBSD repository catalogue..
+FreeBSD repository is up-to-date.
+All repositories are up-to-date.
+The following 1 package(s) will be affected (of 0 checked):
+
+New packages to be INSTALLED:
+        go: 1.6.2,1
+        gmake: 4.1_2
+
+        The process will require 261 MiB more space.
+        40 MiB to be downloaded.
+
+        Proceed with this action? [y/N]: y
+        Fetching go-1.6.2,1.txz: 100%   40 MiB  10.4MB/s    00:04
+        Fetching gmake-4.1_2.txz: 100%  363 KiB 372.2kB/s    00:01
+        Checking integrity... done (0 conflicting)
+        [1/2] Installing go-1.6.2,1...
+        [1/2] Extracting go-1.6.2,1: 100%
+        [2/2] Installing gmake-4.1_2...
+        [2/2] Extracting gmake-4.1_2: 100%
+```
+
+Next we need to configure our environment for golang.  Insert the following lines into your ~/.profile file, and be sure to source the file before proceeding to the next step:
+
+```
+GOPATH=$HOME/golang; export GOPATH
+GOROOT=/usr/local/go/; export GOROOT
+```
+
+Now we can proceed with the normal build process of minio server as found [here](https://github.com/nomadlogic/minio/blob/master/CONTRIBUTING.md).  The only caveat is we need to specify gmake (GNU make) when building minio server as the current Makefile is not BSD make compatible:
+
+```
+$ mkdir -p $GOPATH/src/github.com/minio
+$ cd $GOPATH/src/github.com/minio
+$ git clone <paste saved URL for personal forked minio repo>
+$ cd minio
+$ gmake
+```
+
+From here you can start the server as you would with a precompiled minio server build.

Makefile

@@ -71,6 +71,7 @@ getdeps: checks
 	@go get -u github.com/fzipp/gocyclo && echo "Installed gocyclo:"
 	@go get -u github.com/remyoudompheng/go-misc/deadcode && echo "Installed deadcode:"
 	@go get -u github.com/client9/misspell/cmd/misspell && echo "Installed misspell:"
+	@go get -u github.com/gordonklaus/ineffassign && echo "Installed ineffassign:"
 
 verifiers: vet fmt lint cyclo spelling
 
@@ -91,6 +92,10 @@ lint:
 	@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/golint *.go
 	@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/golint github.com/minio/minio/pkg...
 
+ineffassign:
+	@echo "Running $@:"
+	@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/ineffassign .
+
 cyclo:
 	@echo "Running $@:"
 	@GO15VENDOREXPERIMENT=1 ${GOPATH}/bin/gocyclo -over 65 *.go

Multipart.md

@@ -1,10 +0,0 @@
-## Multipart backend format
-
-When multipart upload is used for objects, below meta-data/staging files are created 
-
-- New multipart upload call creates file ```EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.uploadid```
-- Put object part call creates file ```EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.PART_NUMBER.MD5SUM_STRING```
-- Abort multipart call removes all files matching ```EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.*```
-- Complete multipart call does
-  1. Create a staging file ```EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete.TEMP_NAME``` then rename to ```EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete```
-  2. Rename staging file ```EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete``` to ```EXPORT_DIR/BUCKET/PATH/TO/OBJECT```

README.md

@@ -9,8 +9,8 @@ Minio is light enough to be bundled with the application stack. It sits on the s
 
 Minio currently implements two backends
 
-  - Filesystem (FS) - is available and ready for general purpose use. This version of the Minio binary is built using Filesystem storage backend for magnetic and solid state disks.
-  - ErasureCoded (XL) - is work in progress and not ready for general purpose use.
+  - Filesystem (FS) - is available and ready for general purpose use.
+  - ErasureCoded (XL) - is available, but it is not ready for general purpose use.
 
 ## Minio Client
 
@@ -66,6 +66,7 @@ Download ``minio`` from https://dl.minio.io/server/minio/release/freebsd-amd64/m
 $ chmod 755 minio
 $ ./minio --help
 ~~~
+Read more here on [How to configure Minio on FreeBSD with ZFS backend.](./FreeBSD.md)
 
 #### Docker container
 
@@ -79,7 +80,7 @@ Read more here on [How to configure data volume containers for Minio?](./Docker.
 
 #### Source
 <blockquote>
-NOTE: Source installation is intended for only developers and advanced users.  For general use, please download official releases from https://minio.io/download.
+NOTE: Source installation is intended for only developers and advanced users.  For general use, please download official releases from https://minio.io/downloads.
 </blockquote>
 
 If you do not have a working Golang environment, please follow [Install Golang](./INSTALLGO.md).
@@ -138,16 +139,41 @@ $ aws configure set default.s3.signature_version s3v4
 To list your buckets.
 ```
 $ aws --endpoint-url http://localhost:9000 s3 ls
-2016-01-07 16:38:23 testbucket
+2016-03-27 02:06:30 deebucket
+2016-03-28 21:53:49 guestbucket
+2016-03-29 13:34:34 mbtest
+2016-03-26 22:01:36 mybucket
+2016-03-26 15:37:02 testbucket
 ```
 
 To list contents inside bucket.
 ```
-$ aws --endpoint-url http://localhost:9000 s3 ls s3://testbucket
-                           PRE test/
-2015-12-17 08:46:41   12232928 vim
-2016-01-07 16:38:23   32232928 emacs
-2015-12-09 08:05:24     138504 s3cmd
+$ aws --endpoint-url http://localhost:9000 s3 ls s3://mybucket
+2016-03-30 00:26:53      69297 argparse-1.2.1.tar.gz
+2016-03-30 00:35:37      67250 simplejson-3.3.0.tar.gz
+```
+
+To make a bucket.
+```
+$ aws --endpoint-url http://localhost:9000 s3 mb s3://mybucket
+make_bucket: s3://mybucket/
+```
+
+To add an object to a bucket.
+```
+$ aws --endpoint-url http://localhost:9000 s3 cp simplejson-3.3.0.tar.gz s3://mybucket
+upload: ./simplejson-3.3.0.tar.gz to s3://mybucket/simplejson-3.3.0.tar.gz
+```
+
+Delete an object from a bucket.
+```
+$ aws --endpoint-url http://localhost:9000 s3 rm s3://mybucket/argparse-1.2.1.tar.gzdelete: s3://mybucket/argparse-1.2.1.tar.gz
+```
+
+Remove a bucket.
+```
+$ aws --endpoint-url http://localhost:9000 s3 rb s3://mybucket
+remove_bucket: s3://mybucket/
 ```
 
 #### How to use AWS SDK with Minio?
@@ -168,36 +194,37 @@ host_bucket = localhost:9000
 access_key = YOUR_ACCESS_KEY_HERE
 secret_key = YOUR_SECRET_KEY_HERE
 signature_v2 = False
+bucket_location = us-east-1
 ```
 
-To make a bucket
+To make a bucket.
 ```
 $ s3cmd mb s3://mybucket
 Bucket 's3://mybucket/' created
 ```
 
-To copy an object to bucket
+To copy an object to bucket.
 ```
-$ s3cmd put newfile.txt s3://testbucket
+$ s3cmd put newfile s3://testbucket
 upload: 'newfile' -> 's3://testbucket/newfile'  
 ```
 
-To copy an object to local system
+To copy an object to local system.
 ```
 $ s3cmd get s3://testbucket/newfile
 download: 's3://testbucket/newfile' -> './newfile'
 ```
 
-To sync local file/directory to a bucket 
+To sync local file/directory to a bucket.
 ```
 $ s3cmd sync newdemo s3://testbucket
 upload: 'newdemo/newdemofile.txt' -> 's3://testbucket/newdemo/newdemofile.txt'
 ```
 
-To sync bucket or object with local filesystem
+To sync bucket or object with local filesystem.
 ```
 $ s3cmd sync  s3://otherbucket otherlocalbucket
-download: 's3://otherbucket/cat.jpg' -> 'otherlocalbucket/cat.jpg' 
+download: 's3://otherbucket/cat.jpg' -> 'otherlocalbucket/cat.jpg'
 ```
 
 To list buckets.
@@ -213,13 +240,13 @@ $ s3cmd ls s3://testbucket/
 2015-12-09 16:05    138504   s3://testbucket/newfile
 ```
 
-Delete an object from bucket
+Delete an object from bucket.
 ```
 $ s3cmd del s3://testbucket/newfile
 delete: 's3://testbucket/newfile'
 ```
 
-Delete a bucket
+Delete a bucket.
 ```
 $ s3cmd rb s3://testbucket
 Bucket 's3://testbucket/' removed

access-key.go

@@ -21,8 +21,6 @@ import (
 	"encoding/base64"
 	"fmt"
 	"regexp"
-
-	"github.com/minio/minio/pkg/probe"
 )
 
 // credential container for access and secret keys.
@@ -52,19 +50,19 @@ var isValidAccessKey = regexp.MustCompile(`^[a-zA-Z0-9\\-\\.\\_\\~]{5,20}$`)
 // mustGenAccessKeys - must generate access credentials.
 func mustGenAccessKeys() (creds credential) {
 	creds, err := genAccessKeys()
-	fatalIf(err.Trace(), "Unable to generate access keys.", nil)
+	fatalIf(err, "Unable to generate access keys.")
 	return creds
 }
 
 // genAccessKeys - generate access credentials.
-func genAccessKeys() (credential, *probe.Error) {
+func genAccessKeys() (credential, error) {
 	accessKeyID, err := genAccessKeyID()
 	if err != nil {
-		return credential{}, err.Trace()
+		return credential{}, err
 	}
 	secretAccessKey, err := genSecretAccessKey()
 	if err != nil {
-		return credential{}, err.Trace()
+		return credential{}, err
 	}
 	creds := credential{
 		AccessKeyID:     string(accessKeyID),
@@ -75,10 +73,10 @@ func genAccessKeys() (credential, *probe.Error) {
 
 // genAccessKeyID - generate random alpha numeric value using only uppercase characters
 // takes input as size in integer
-func genAccessKeyID() ([]byte, *probe.Error) {
+func genAccessKeyID() ([]byte, error) {
 	alpha := make([]byte, minioAccessID)
-	if _, e := rand.Read(alpha); e != nil {
-		return nil, probe.NewError(e)
+	if _, err := rand.Read(alpha); err != nil {
+		return nil, err
 	}
 	for i := 0; i < minioAccessID; i++ {
 		alpha[i] = alphaNumericTable[alpha[i]%byte(len(alphaNumericTable))]
@@ -87,10 +85,10 @@ func genAccessKeyID() ([]byte, *probe.Error) {
 }
 
 // genSecretAccessKey - generate random base64 numeric value from a random seed.
-func genSecretAccessKey() ([]byte, *probe.Error) {
+func genSecretAccessKey() ([]byte, error) {
 	rb := make([]byte, minioSecretID)
-	if _, e := rand.Read(rb); e != nil {
-		return nil, probe.NewError(e)
+	if _, err := rand.Read(rb); err != nil {
+		return nil, err
 	}
 	return []byte(base64.StdEncoding.EncodeToString(rb))[:minioSecretID], nil
 }

accesslog-handler.go

@@ -22,8 +22,6 @@ import (
 	"net/url"
 	"os"
 	"time"
-
-	"github.com/minio/minio/pkg/probe"
 )
 
 type accessLogHandler struct {
@@ -60,14 +58,14 @@ type LogMessage struct {
 
 func (h *accessLogHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 	message, err := getLogMessage(w, req)
-	fatalIf(err.Trace(), "Unable to extract http message.", nil)
-	_, e := h.accessLogFile.Write(message)
-	fatalIf(probe.NewError(e), "Writing to log file failed.", nil)
+	fatalIf(err, "Unable to parse HTTP request and response fields.")
+	_, err = h.accessLogFile.Write(message)
+	fatalIf(err, "Unable to log HTTP access.")
 
 	h.Handler.ServeHTTP(w, req)
 }
 
-func getLogMessage(w http.ResponseWriter, req *http.Request) ([]byte, *probe.Error) {
+func getLogMessage(w http.ResponseWriter, req *http.Request) ([]byte, error) {
 	logMessage := &LogMessage{
 		StartTime: time.Now().UTC(),
 	}
@@ -103,9 +101,9 @@ func getLogMessage(w http.ResponseWriter, req *http.Request) ([]byte, *probe.Err
 
 	// logMessage.HTTP.Request = req
 	logMessage.Duration = time.Now().UTC().Sub(logMessage.StartTime)
-	js, e := json.Marshal(logMessage)
-	if e != nil {
-		return nil, probe.NewError(e)
+	js, err := json.Marshal(logMessage)
+	if err != nil {
+		return nil, err
 	}
 	js = append(js, byte('\n')) // append a new line
 	return js, nil
@@ -113,8 +111,8 @@ func getLogMessage(w http.ResponseWriter, req *http.Request) ([]byte, *probe.Err
 
 // setAccessLogHandler logs requests
 func setAccessLogHandler(h http.Handler) http.Handler {
-	file, e := os.OpenFile("access.log", os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
-	fatalIf(probe.NewError(e), "Unable to open access log.", nil)
+	file, err := os.OpenFile("access.log", os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
+	fatalIf(err, "Failed top open access log.")
 
 	return &accessLogHandler{Handler: h, accessLogFile: file}
 }

api-datatypes.go

@@ -16,11 +16,22 @@
 
 package main
 
+import (
+	"encoding/xml"
+)
+
 // ObjectIdentifier carries key name for the object to delete.
 type ObjectIdentifier struct {
 	ObjectName string `xml:"Key"`
 }
 
+// createBucketConfiguration container for bucket configuration request from client.
+// Used for parsing the location from the request body for MakeBucketbucket.
+type createBucketLocationConfiguration struct {
+	XMLName  xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CreateBucketConfiguration" json:"-"`
+	Location string   `xml:"LocationConstraint"`
+}
+
 // DeleteObjectsRequest - xml carrying the object key names which needs to be deleted.
 type DeleteObjectsRequest struct {
 	// Element to enable quiet mode for the request

api-errors.go

@@ -83,8 +83,6 @@ const (
 	ErrMalformedPOSTRequest
 	ErrSignatureVersionNotSupported
 	ErrBucketNotEmpty
-	ErrRootPathFull
-	ErrObjectExistsAsPrefix
 	ErrAllAccessDisabled
 	ErrMalformedPolicy
 	ErrMissingFields
@@ -99,12 +97,18 @@ const (
 	ErrMalformedDate
 	ErrMalformedExpires
 	ErrAuthHeaderEmpty
-	ErrDateHeaderMissing
 	ErrExpiredPresignRequest
 	ErrMissingDateHeader
 	ErrInvalidQuerySignatureAlgo
 	ErrInvalidQueryParams
+	ErrBucketAlreadyOwnedByYou
 	// Add new error codes here.
+
+	// Minio extended errors.
+	ErrReadQuorum
+	ErrWriteQuorum
+	ErrStorageFull
+	ErrObjectExistsAsDirectory
 )
 
 // error code to APIError structure, these fields carry respective
@@ -295,16 +299,6 @@ var errorCodeResponse = map[APIErrorCode]APIError{
 		Description:    "The bucket you tried to delete is not empty.",
 		HTTPStatusCode: http.StatusConflict,
 	},
-	ErrRootPathFull: {
-		Code:           "RootPathFull",
-		Description:    "Root path has reached its minimum free disk threshold. Please delete few objects to proceed.",
-		HTTPStatusCode: http.StatusInternalServerError,
-	},
-	ErrObjectExistsAsPrefix: {
-		Code:           "ObjectExistsAsPrefix",
-		Description:    "An object already exists as your prefix, choose a different prefix to proceed.",
-		HTTPStatusCode: http.StatusConflict,
-	},
 	ErrAllAccessDisabled: {
 		Code:           "AllAccessDisabled",
 		Description:    "All access to this bucket has been disabled.",
@@ -395,9 +389,83 @@ var errorCodeResponse = map[APIErrorCode]APIError{
 		Description:    "Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.",
 		HTTPStatusCode: http.StatusBadRequest,
 	},
+	ErrBucketAlreadyOwnedByYou: {
+		Code:           "BucketAlreadyOwnedByYou",
+		Description:    "Your previous request to create the named bucket succeeded and you already own it.",
+		HTTPStatusCode: http.StatusConflict,
+	},
+	/// Minio extensions.
+	ErrStorageFull: {
+		Code:           "XMinioStorageFull",
+		Description:    "Storage backend has reached its minimum free disk threshold. Please delete few objects to proceed.",
+		HTTPStatusCode: http.StatusInternalServerError,
+	},
+	ErrObjectExistsAsDirectory: {
+		Code:           "XMinioObjectExistsAsDirectory",
+		Description:    "Object name already exists as a directory.",
+		HTTPStatusCode: http.StatusConflict,
+	},
+	ErrReadQuorum: {
+		Code:           "XMinioReadQuorum",
+		Description:    "Multiple disk failures, unable to reconstruct data.",
+		HTTPStatusCode: http.StatusServiceUnavailable,
+	},
+	ErrWriteQuorum: {
+		Code:           "XMinioWriteQuorum",
+		Description:    "Multiple disks failures, unable to write data.",
+		HTTPStatusCode: http.StatusServiceUnavailable,
+	},
 	// Add your error structure here.
 }
 
+// toAPIErrorCode - Converts embedded errors. Convenience
+// function written to handle all cases where we have known types of
+// errors returned by underlying layers.
+func toAPIErrorCode(err error) (apiErr APIErrorCode) {
+	if err == nil {
+		return ErrNone
+	}
+	// Verify if the underlying error is signature mismatch.
+	if err == errSignatureMismatch {
+		return ErrSignatureDoesNotMatch
+	}
+	switch err.(type) {
+	case StorageFull:
+		apiErr = ErrStorageFull
+	case BadDigest:
+		apiErr = ErrBadDigest
+	case IncompleteBody:
+		apiErr = ErrIncompleteBody
+	case ObjectExistsAsDirectory:
+		apiErr = ErrObjectExistsAsDirectory
+	case BucketNameInvalid:
+		apiErr = ErrInvalidBucketName
+	case BucketNotFound:
+		apiErr = ErrNoSuchBucket
+	case BucketNotEmpty:
+		apiErr = ErrBucketNotEmpty
+	case BucketExists:
+		apiErr = ErrBucketAlreadyOwnedByYou
+	case ObjectNotFound:
+		apiErr = ErrNoSuchKey
+	case ObjectNameInvalid:
+		apiErr = ErrNoSuchKey
+	case InvalidUploadID:
+		apiErr = ErrNoSuchUpload
+	case InvalidPart:
+		apiErr = ErrInvalidPart
+	case InsufficientWriteQuorum:
+		apiErr = ErrWriteQuorum
+	case InsufficientReadQuorum:
+		apiErr = ErrReadQuorum
+	case PartTooSmall:
+		apiErr = ErrEntityTooSmall
+	default:
+		apiErr = ErrInternalError
+	}
+	return apiErr
+}
+
 // getAPIError provides API Error for input API error code.
 func getAPIError(code APIErrorCode) APIError {
 	return errorCodeResponse[code]

api-headers.go

@@ -63,7 +63,7 @@ func setObjectHeaders(w http.ResponseWriter, objInfo ObjectInfo, contentRange *h
 	setCommonHeaders(w)
 
 	// set object-related metadata headers
-	lastModified := objInfo.ModifiedTime.UTC().Format(http.TimeFormat)
+	lastModified := objInfo.ModTime.UTC().Format(http.TimeFormat)
 	w.Header().Set("Last-Modified", lastModified)
 
 	w.Header().Set("Content-Type", objInfo.ContentType)

api-resources.go

@@ -22,7 +22,39 @@ import (
 )
 
 // Parse bucket url queries
-func getBucketResources(values url.Values) (prefix, marker, delimiter string, maxkeys int, encodingType string) {
+func getListObjectsV1Args(values url.Values) (prefix, marker, delimiter string, maxkeys int, encodingType string) {
+	prefix = values.Get("prefix")
+	marker = values.Get("marker")
+	delimiter = values.Get("delimiter")
+	if values.Get("max-keys") != "" {
+		maxkeys, _ = strconv.Atoi(values.Get("max-keys"))
+	} else {
+		maxkeys = maxObjectList
+	}
+	encodingType = values.Get("encoding-type")
+	return
+}
+
+// Parse bucket url queries for ListObjects V2.
+func getListObjectsV2Args(values url.Values) (prefix, token, startAfter, delimiter string, maxkeys int, encodingType string) {
+	prefix = values.Get("prefix")
+	startAfter = values.Get("start-after")
+	delimiter = values.Get("delimiter")
+	if values.Get("max-keys") != "" {
+		maxkeys, _ = strconv.Atoi(values.Get("max-keys"))
+	} else {
+		maxkeys = maxObjectList
+	}
+	encodingType = values.Get("encoding-type")
+	token = values.Get("continuation-token")
+	return
+}
+
+// Parse bucket url queries
+func getBucketResources(values url.Values) (listType int, prefix, marker, delimiter string, maxkeys int, encodingType string) {
+	if values.Get("list-type") != "" {
+		listType, _ = strconv.Atoi(values.Get("list-type"))
+	}
 	prefix = values.Get("prefix")
 	marker = values.Get("marker")
 	delimiter = values.Get("delimiter")
@@ -62,9 +94,3 @@ func getObjectResources(values url.Values) (uploadID string, partNumberMarker, m
 	encodingType = values.Get("encoding-type")
 	return
 }
-
-// Get upload id.
-func getUploadID(values url.Values) (uploadID string) {
-	uploadID, _, _, _ = getObjectResources(values)
-	return
-}

api-response.go

@@ -65,6 +65,37 @@ type ListObjectsResponse struct {
 	Prefix     string
 }
 
+// ListObjectsV2Response - format for list objects response.
+type ListObjectsV2Response struct {
+	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListBucketResult" json:"-"`
+
+	CommonPrefixes []CommonPrefix
+	Contents       []Object
+
+	Delimiter string
+
+	// Encoding type used to encode object keys in the response.
+	EncodingType string
+
+	// A flag that indicates whether or not ListObjects returned all of the results
+	// that satisfied the search criteria.
+	IsTruncated bool
+	StartAfter  string
+	MaxKeys     int
+	Name        string
+
+	// When response is truncated (the IsTruncated element value in the response
+	// is true), you can use the key name in this field as marker in the subsequent
+	// request to get next set of objects. Server lists objects in alphabetical
+	// order Note: This element is returned only if you have delimiter request parameter
+	// specified. If response does not include the NextMaker and it is truncated,
+	// you can use the value of the last Key in the response as the marker in the
+	// subsequent request to get the next set of object keys.
+	ContinuationToken     string
+	NextContinuationToken string
+	Prefix                string
+}
+
 // Part container for part metadata.
 type Part struct {
 	PartNumber   int
@@ -194,6 +225,16 @@ type CompleteMultipartUploadResponse struct {
 	ETag     string
 }
 
+// PostResponse container for completed post upload response
+type PostResponse struct {
+	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ PostResponse" json:"-"`
+
+	Location string
+	Bucket   string
+	Key      string
+	ETag     string
+}
+
 // DeleteError structure.
 type DeleteError struct {
 	Code    string
@@ -217,6 +258,11 @@ func getLocation(r *http.Request) string {
 	return r.URL.Path
 }
 
+// getObjectLocation gets the relative URL for an object
+func getObjectLocation(bucketName string, key string) string {
+	return "/" + bucketName + "/" + key
+}
+
 // takes an array of Bucketmetadata information for serialization
 // input:
 // array of bucket metadata
@@ -260,7 +306,7 @@ func generateListObjectsResponse(bucket, prefix, marker, delimiter string, maxKe
 			continue
 		}
 		content.Key = object.Name
-		content.LastModified = object.ModifiedTime.UTC().Format(timeFormatAMZ)
+		content.LastModified = object.ModTime.UTC().Format(timeFormatAMZ)
 		if object.MD5Sum != "" {
 			content.ETag = "\"" + object.MD5Sum + "\""
 		}
@@ -289,6 +335,51 @@ func generateListObjectsResponse(bucket, prefix, marker, delimiter string, maxKe
 	return data
 }
 
+// generates an ListObjects response for the said bucket with other enumerated options.
+func generateListObjectsV2Response(bucket, prefix, token, startAfter, delimiter string, maxKeys int, resp ListObjectsInfo) ListObjectsV2Response {
+	var contents []Object
+	var prefixes []CommonPrefix
+	var owner = Owner{}
+	var data = ListObjectsV2Response{}
+
+	owner.ID = "minio"
+	owner.DisplayName = "minio"
+
+	for _, object := range resp.Objects {
+		var content = Object{}
+		if object.Name == "" {
+			continue
+		}
+		content.Key = object.Name
+		content.LastModified = object.ModTime.UTC().Format(timeFormatAMZ)
+		if object.MD5Sum != "" {
+			content.ETag = "\"" + object.MD5Sum + "\""
+		}
+		content.Size = object.Size
+		content.StorageClass = "STANDARD"
+		content.Owner = owner
+		contents = append(contents, content)
+	}
+	// TODO - support EncodingType in xml decoding
+	data.Name = bucket
+	data.Contents = contents
+
+	data.StartAfter = startAfter
+	data.Delimiter = delimiter
+	data.Prefix = prefix
+	data.MaxKeys = maxKeys
+	data.ContinuationToken = token
+	data.NextContinuationToken = resp.NextMarker
+	data.IsTruncated = resp.IsTruncated
+	for _, prefix := range resp.Prefixes {
+		var prefixItem = CommonPrefix{}
+		prefixItem.Prefix = prefix
+		prefixes = append(prefixes, prefixItem)
+	}
+	data.CommonPrefixes = prefixes
+	return data
+}
+
 // generateCopyObjectResponse
 func generateCopyObjectResponse(etag string, lastModified time.Time) CopyObjectResponse {
 	return CopyObjectResponse{

api-router.go

@@ -18,13 +18,13 @@ package main
 
 import router "github.com/gorilla/mux"
 
-// objectStorageAPI container for S3 compatible API.
-type objectStorageAPI struct {
-	ObjectAPI ObjectAPI
+// objectAPIHandler implements and provides http handlers for S3 API.
+type objectAPIHandlers struct {
+	ObjectAPI ObjectLayer
 }
 
 // registerAPIRouter - registers S3 compatible APIs.
-func registerAPIRouter(mux *router.Router, api objectStorageAPI) {
+func registerAPIRouter(mux *router.Router, api objectAPIHandlers) {
 	// API Router
 	apiRouter := mux.NewRoute().PathPrefix("/").Subrouter()
 

auth-handler.go

@@ -19,14 +19,12 @@ package main
 import (
 	"bytes"
 	"crypto/md5"
+	"crypto/sha256"
 	"encoding/base64"
 	"encoding/hex"
 	"io/ioutil"
 	"net/http"
 	"strings"
-
-	fastSha256 "github.com/minio/minio/pkg/crypto/sha256"
-	"github.com/minio/minio/pkg/probe"
 )
 
 // Verify if request has JWT.
@@ -98,7 +96,7 @@ func getRequestAuthType(r *http.Request) authType {
 
 // sum256 calculate sha256 sum for an input byte array
 func sum256(data []byte) []byte {
-	hash := fastSha256.New()
+	hash := sha256.New()
 	hash.Write(data)
 	return hash.Sum(nil)
 }
@@ -113,12 +111,10 @@ func sumMD5(data []byte) []byte {
 // Verify if request has valid AWS Signature Version '4'.
 func isReqAuthenticated(r *http.Request) (s3Error APIErrorCode) {
 	if r == nil {
-		errorIf(probe.NewError(errInvalidArgument), "HTTP request cannot be empty.", nil)
 		return ErrInternalError
 	}
-	payload, e := ioutil.ReadAll(r.Body)
-	if e != nil {
-		errorIf(probe.NewError(e), "Unable to read HTTP body.", nil)
+	payload, err := ioutil.ReadAll(r.Body)
+	if err != nil {
 		return ErrInternalError
 	}
 	// Verify Content-Md5, if payload is set.

bucket-handlers.go

@@ -29,7 +29,6 @@ import (
 	"strings"
 
 	mux "github.com/gorilla/mux"
-	"github.com/minio/minio/pkg/probe"
 )
 
 // http://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
@@ -37,8 +36,8 @@ func enforceBucketPolicy(action string, bucket string, reqURL *url.URL) (s3Error
 	// Read saved bucket policy.
 	policy, err := readBucketPolicy(bucket)
 	if err != nil {
-		errorIf(err.Trace(bucket), "GetBucketPolicy failed.", nil)
-		switch err.ToGoError().(type) {
+		errorIf(err, "Unable read bucket policy.")
+		switch err.(type) {
 		case BucketNotFound:
 			return ErrNoSuchBucket
 		case BucketNameInvalid:
@@ -49,13 +48,13 @@ func enforceBucketPolicy(action string, bucket string, reqURL *url.URL) (s3Error
 		}
 	}
 	// Parse the saved policy.
-	bucketPolicy, e := parseBucketPolicy(policy)
-	if e != nil {
-		errorIf(probe.NewError(e), "Parse policy failed.", nil)
+	bucketPolicy, err := parseBucketPolicy(policy)
+	if err != nil {
+		errorIf(err, "Unable to parse bucket policy.")
 		return ErrAccessDenied
 	}
 
-	// Construct resource in 'arn:aws:s3:::examplebucket' format.
+	// Construct resource in 'arn:aws:s3:::examplebucket/object' format.
 	resource := AWSResourcePrefix + strings.TrimPrefix(reqURL.Path, "/")
 
 	// Get conditions for policy verification.
@@ -74,7 +73,7 @@ func enforceBucketPolicy(action string, bucket string, reqURL *url.URL) (s3Error
 // GetBucketLocationHandler - GET Bucket location.
 // -------------------------
 // This operation returns bucket location.
-func (api objectStorageAPI) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -90,8 +89,8 @@ func (api objectStorageAPI) GetBucketLocationHandler(w http.ResponseWriter, r *h
 			return
 		}
 	case authTypeSigned, authTypePresigned:
-		payload, e := ioutil.ReadAll(r.Body)
-		if e != nil {
+		payload, err := ioutil.ReadAll(r.Body)
+		if err != nil {
 			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
 			return
 		}
@@ -117,17 +116,9 @@ func (api objectStorageAPI) GetBucketLocationHandler(w http.ResponseWriter, r *h
 		}
 	}
 
-	_, err := api.ObjectAPI.GetBucketInfo(bucket)
-	if err != nil {
-		errorIf(err.Trace(), "GetBucketInfo failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+	if _, err := api.ObjectAPI.GetBucketInfo(bucket); err != nil {
+		errorIf(err, "Unable to fetch bucket info.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 
@@ -152,7 +143,7 @@ func (api objectStorageAPI) GetBucketLocationHandler(w http.ResponseWriter, r *h
 // completed or aborted. This operation returns at most 1,000 multipart
 // uploads in the response.
 //
-func (api objectStorageAPI) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -180,32 +171,17 @@ func (api objectStorageAPI) ListMultipartUploadsHandler(w http.ResponseWriter, r
 		return
 	}
 	if keyMarker != "" {
-		// Unescape keyMarker string
-		keyMarkerUnescaped, e := url.QueryUnescape(keyMarker)
-		if e != nil {
-			if e != nil {
-				// Return 'NoSuchKey' to indicate invalid marker key.
-				writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-				return
-			}
-			keyMarker = keyMarkerUnescaped
-			// Marker not common with prefix is not implemented.
-			if !strings.HasPrefix(keyMarker, prefix) {
-				writeErrorResponse(w, r, ErrNotImplemented, r.URL.Path)
-				return
-			}
+		// Marker not common with prefix is not implemented.
+		if !strings.HasPrefix(keyMarker, prefix) {
+			writeErrorResponse(w, r, ErrNotImplemented, r.URL.Path)
+			return
 		}
 	}
 
 	listMultipartsInfo, err := api.ObjectAPI.ListMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
 	if err != nil {
-		errorIf(err.Trace(), "ListMultipartUploads failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		errorIf(err, "Unable to list multipart uploads.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 	// generate response
@@ -223,7 +199,7 @@ func (api objectStorageAPI) ListMultipartUploadsHandler(w http.ResponseWriter, r
 // of the objects in a bucket. You can use the request parameters as selection
 // criteria to return a subset of the objects in a bucket.
 //
-func (api objectStorageAPI) ListObjectsHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) ListObjectsHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -244,9 +220,22 @@ func (api objectStorageAPI) ListObjectsHandler(w http.ResponseWriter, r *http.Re
 			return
 		}
 	}
-
+	var prefix, marker, token, delimiter, startAfter string
+	var maxkeys int
+	var listV2 bool
 	// TODO handle encoding type.
-	prefix, marker, delimiter, maxkeys, _ := getBucketResources(r.URL.Query())
+	if r.URL.Query().Get("list-type") == "2" {
+		listV2 = true
+		prefix, token, startAfter, delimiter, maxkeys, _ = getListObjectsV2Args(r.URL.Query())
+		// For ListV2 "start-after" is considered only if "continuation-token" is empty.
+		if token == "" {
+			marker = startAfter
+		} else {
+			marker = token
+		}
+	} else {
+		prefix, marker, delimiter, maxkeys, _ = getListObjectsV1Args(r.URL.Query())
+	}
 	if maxkeys < 0 {
 		writeErrorResponse(w, r, ErrInvalidMaxKeys, r.URL.Path)
 		return
@@ -258,14 +247,6 @@ func (api objectStorageAPI) ListObjectsHandler(w http.ResponseWriter, r *http.Re
 	}
 	// If marker is set unescape.
 	if marker != "" {
-		// Try to unescape marker.
-		markerUnescaped, e := url.QueryUnescape(marker)
-		if e != nil {
-			// Return 'NoSuchKey' to indicate invalid marker key.
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-			return
-		}
-		marker = markerUnescaped
 		// Marker not common with prefix is not implemented.
 		if !strings.HasPrefix(marker, prefix) {
 			writeErrorResponse(w, r, ErrNotImplemented, r.URL.Path)
@@ -274,36 +255,32 @@ func (api objectStorageAPI) ListObjectsHandler(w http.ResponseWriter, r *http.Re
 	}
 
 	listObjectsInfo, err := api.ObjectAPI.ListObjects(bucket, prefix, marker, delimiter, maxkeys)
+
 	if err == nil {
+		var encodedSuccessResponse []byte
 		// generate response
-		response := generateListObjectsResponse(bucket, prefix, marker, delimiter, maxkeys, listObjectsInfo)
-		encodedSuccessResponse := encodeResponse(response)
+		if listV2 {
+			response := generateListObjectsV2Response(bucket, prefix, token, startAfter, delimiter, maxkeys, listObjectsInfo)
+			encodedSuccessResponse = encodeResponse(response)
+		} else {
+			response := generateListObjectsResponse(bucket, prefix, marker, delimiter, maxkeys, listObjectsInfo)
+			encodedSuccessResponse = encodeResponse(response)
+		}
 		// Write headers
 		setCommonHeaders(w)
 		// Write success response.
 		writeSuccessResponse(w, encodedSuccessResponse)
 		return
 	}
-	switch err.ToGoError().(type) {
-	case BucketNameInvalid:
-		writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-	case BucketNotFound:
-		writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-	case ObjectNotFound:
-		writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-	case ObjectNameInvalid:
-		writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-	default:
-		errorIf(err.Trace(), "ListObjects failed.", nil)
-		writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-	}
+	errorIf(err, "Unable to list objects.")
+	writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 }
 
 // ListBucketsHandler - GET Service
 // -----------
 // This implementation of the GET operation returns a list of all buckets
 // owned by the authenticated sender of the request.
-func (api objectStorageAPI) ListBucketsHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.Request) {
 	// List buckets does not support bucket policies.
 	switch getRequestAuthType(r) {
 	default:
@@ -349,12 +326,12 @@ func (api objectStorageAPI) ListBucketsHandler(w http.ResponseWriter, r *http.Re
 		writeSuccessResponse(w, encodedSuccessResponse)
 		return
 	}
-	errorIf(err.Trace(), "ListBuckets failed.", nil)
-	writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
+	errorIf(err, "Unable to list buckets.")
+	writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 }
 
 // DeleteMultipleObjectsHandler - deletes multiple objects.
-func (api objectStorageAPI) DeleteMultipleObjectsHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -394,16 +371,16 @@ func (api objectStorageAPI) DeleteMultipleObjectsHandler(w http.ResponseWriter,
 	deleteXMLBytes := make([]byte, r.ContentLength)
 
 	// Read incoming body XML bytes.
-	_, e := io.ReadFull(r.Body, deleteXMLBytes)
-	if e != nil {
-		errorIf(probe.NewError(e), "DeleteMultipleObjects failed.", nil)
+	if _, err := io.ReadFull(r.Body, deleteXMLBytes); err != nil {
+		errorIf(err, "Unable to read HTTP body.")
 		writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
 		return
 	}
 
 	// Unmarshal list of keys to be deleted.
 	deleteObjects := &DeleteObjectsRequest{}
-	if e := xml.Unmarshal(deleteXMLBytes, deleteObjects); e != nil {
+	if err := xml.Unmarshal(deleteXMLBytes, deleteObjects); err != nil {
+		errorIf(err, "Unable to unmarshal delete objects request XML.")
 		writeErrorResponse(w, r, ErrMalformedXML, r.URL.Path)
 		return
 	}
@@ -418,39 +395,12 @@ func (api objectStorageAPI) DeleteMultipleObjectsHandler(w http.ResponseWriter,
 				ObjectName: object.ObjectName,
 			})
 		} else {
-			errorIf(err.Trace(object.ObjectName), "DeleteObject failed.", nil)
-			switch err.ToGoError().(type) {
-			case BucketNameInvalid:
-				deleteErrors = append(deleteErrors, DeleteError{
-					Code:    errorCodeResponse[ErrInvalidBucketName].Code,
-					Message: errorCodeResponse[ErrInvalidBucketName].Description,
-					Key:     object.ObjectName,
-				})
-			case BucketNotFound:
-				deleteErrors = append(deleteErrors, DeleteError{
-					Code:    errorCodeResponse[ErrNoSuchBucket].Code,
-					Message: errorCodeResponse[ErrNoSuchBucket].Description,
-					Key:     object.ObjectName,
-				})
-			case ObjectNotFound:
-				deleteErrors = append(deleteErrors, DeleteError{
-					Code:    errorCodeResponse[ErrNoSuchKey].Code,
-					Message: errorCodeResponse[ErrNoSuchKey].Description,
-					Key:     object.ObjectName,
-				})
-			case ObjectNameInvalid:
-				deleteErrors = append(deleteErrors, DeleteError{
-					Code:    errorCodeResponse[ErrNoSuchKey].Code,
-					Message: errorCodeResponse[ErrNoSuchKey].Description,
-					Key:     object.ObjectName,
-				})
-			default:
-				deleteErrors = append(deleteErrors, DeleteError{
-					Code:    errorCodeResponse[ErrInternalError].Code,
-					Message: errorCodeResponse[ErrInternalError].Description,
-					Key:     object.ObjectName,
-				})
-			}
+			errorIf(err, "Unable to delete object.")
+			deleteErrors = append(deleteErrors, DeleteError{
+				Code:    errorCodeResponse[toAPIErrorCode(err)].Code,
+				Message: errorCodeResponse[toAPIErrorCode(err)].Description,
+				Key:     object.ObjectName,
+			})
 		}
 	}
 	// Generate response
@@ -465,7 +415,7 @@ func (api objectStorageAPI) DeleteMultipleObjectsHandler(w http.ResponseWriter,
 // PutBucketHandler - PUT Bucket
 // ----------
 // This implementation of the PUT operation creates a new bucket for authenticated request
-func (api objectStorageAPI) PutBucketHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) PutBucketHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -482,18 +432,19 @@ func (api objectStorageAPI) PutBucketHandler(w http.ResponseWriter, r *http.Requ
 		}
 	}
 
+	// the location value in the request body should match the Region in serverConfig.
+	// other values of location are not accepted.
+	// make bucket fails in such cases.
+	errCode := isValidLocationContraint(r.Body, serverConfig.GetRegion())
+	if errCode != ErrNone {
+		writeErrorResponse(w, r, errCode, r.URL.Path)
+		return
+	}
 	// Make bucket.
 	err := api.ObjectAPI.MakeBucket(bucket)
 	if err != nil {
-		errorIf(err.Trace(), "MakeBucket failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketExists:
-			writeErrorResponse(w, r, ErrBucketAlreadyExists, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		errorIf(err, "Unable to create a bucket.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 	// Make sure to add Location information here only for bucket
@@ -501,24 +452,25 @@ func (api objectStorageAPI) PutBucketHandler(w http.ResponseWriter, r *http.Requ
 	writeSuccessResponse(w, nil)
 }
 
-func extractHTTPFormValues(reader *multipart.Reader) (io.Reader, map[string]string, *probe.Error) {
+func extractHTTPFormValues(reader *multipart.Reader) (io.Reader, map[string]string, error) {
 	/// HTML Form values
 	formValues := make(map[string]string)
 	filePart := new(bytes.Buffer)
-	var e error
-	for e == nil {
+	var err error
+	for err == nil {
 		var part *multipart.Part
-		part, e = reader.NextPart()
+		part, err = reader.NextPart()
 		if part != nil {
 			if part.FileName() == "" {
-				buffer, e := ioutil.ReadAll(part)
-				if e != nil {
-					return nil, nil, probe.NewError(e)
+				var buffer []byte
+				buffer, err = ioutil.ReadAll(part)
+				if err != nil {
+					return nil, nil, err
 				}
 				formValues[http.CanonicalHeaderKey(part.FormName())] = string(buffer)
 			} else {
-				if _, e := io.Copy(filePart, part); e != nil {
-					return nil, nil, probe.NewError(e)
+				if _, err = io.Copy(filePart, part); err != nil {
+					return nil, nil, err
 				}
 			}
 		}
@@ -530,19 +482,19 @@ func extractHTTPFormValues(reader *multipart.Reader) (io.Reader, map[string]stri
 // ----------
 // This implementation of the POST operation handles object creation with a specified
 // signature policy in multipart/form-data
-func (api objectStorageAPI) PostPolicyBucketHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *http.Request) {
 	// Here the parameter is the size of the form data that should
 	// be loaded in memory, the remaining being put in temporary files.
-	reader, e := r.MultipartReader()
-	if e != nil {
-		errorIf(probe.NewError(e), "Unable to initialize multipart reader.", nil)
+	reader, err := r.MultipartReader()
+	if err != nil {
+		errorIf(err, "Unable to initialize multipart reader.")
 		writeErrorResponse(w, r, ErrMalformedPOSTRequest, r.URL.Path)
 		return
 	}
 
 	fileBody, formValues, err := extractHTTPFormValues(reader)
 	if err != nil {
-		errorIf(err.Trace(), "Unable to parse form values.", nil)
+		errorIf(err, "Unable to parse form values.")
 		writeErrorResponse(w, r, ErrMalformedPOSTRequest, r.URL.Path)
 		return
 	}
@@ -560,29 +512,28 @@ func (api objectStorageAPI) PostPolicyBucketHandler(w http.ResponseWriter, r *ht
 		writeErrorResponse(w, r, apiErr, r.URL.Path)
 		return
 	}
-	objInfo, err := api.ObjectAPI.PutObject(bucket, object, -1, fileBody, nil)
+
+	// Save metadata.
+	metadata := make(map[string]string)
+	// Nothing to store right now.
+
+	md5Sum, err := api.ObjectAPI.PutObject(bucket, object, -1, fileBody, metadata)
 	if err != nil {
-		errorIf(err.Trace(), "PutObject failed.", nil)
-		switch err.ToGoError().(type) {
-		case RootPathFull:
-			writeErrorResponse(w, r, ErrRootPathFull, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BadDigest:
-			writeErrorResponse(w, r, ErrBadDigest, r.URL.Path)
-		case IncompleteBody:
-			writeErrorResponse(w, r, ErrIncompleteBody, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		errorIf(err, "Unable to create object.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
-	if objInfo.MD5Sum != "" {
-		w.Header().Set("ETag", "\""+objInfo.MD5Sum+"\"")
+	if md5Sum != "" {
+		w.Header().Set("ETag", "\""+md5Sum+"\"")
 	}
-	writeSuccessResponse(w, nil)
+	encodedSuccessResponse := encodeResponse(PostResponse{
+		Location: getObjectLocation(bucket, object), // TODO Full URL is preferred
+		Bucket:   bucket,
+		Key:      object,
+		ETag:     md5Sum,
+	})
+	setCommonHeaders(w)
+	writeSuccessResponse(w, encodedSuccessResponse)
 }
 
 // HeadBucketHandler - HEAD Bucket
@@ -591,7 +542,7 @@ func (api objectStorageAPI) PostPolicyBucketHandler(w http.ResponseWriter, r *ht
 // The operation returns a 200 OK if the bucket exists and you
 // have permission to access it. Otherwise, the operation might
 // return responses such as 404 Not Found and 403 Forbidden.
-func (api objectStorageAPI) HeadBucketHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) HeadBucketHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -613,24 +564,16 @@ func (api objectStorageAPI) HeadBucketHandler(w http.ResponseWriter, r *http.Req
 		}
 	}
 
-	_, err := api.ObjectAPI.GetBucketInfo(bucket)
-	if err != nil {
-		errorIf(err.Trace(), "GetBucketInfo failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+	if _, err := api.ObjectAPI.GetBucketInfo(bucket); err != nil {
+		errorIf(err, "Unable to fetch bucket info.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 	writeSuccessResponse(w, nil)
 }
 
 // DeleteBucketHandler - Delete bucket
-func (api objectStorageAPI) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -646,17 +589,9 @@ func (api objectStorageAPI) DeleteBucketHandler(w http.ResponseWriter, r *http.R
 		}
 	}
 
-	err := api.ObjectAPI.DeleteBucket(bucket)
-	if err != nil {
-		errorIf(err.Trace(), "DeleteBucket failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case BucketNotEmpty:
-			writeErrorResponse(w, r, ErrBucketNotEmpty, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+	if err := api.ObjectAPI.DeleteBucket(bucket); err != nil {
+		errorIf(err, "Unable to delete a bucket.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 

bucket-policy-handlers.go

@@ -25,7 +25,6 @@ import (
 	"strings"
 
 	mux "github.com/gorilla/mux"
-	"github.com/minio/minio/pkg/probe"
 )
 
 // maximum supported access policy size.
@@ -67,8 +66,8 @@ func bucketPolicyMatchStatement(action string, resource string, conditions map[s
 func bucketPolicyActionMatch(action string, statement policyStatement) bool {
 	for _, policyAction := range statement.Actions {
 		// Policy action can be a regex, validate the action with matching string.
-		matched, e := regexp.MatchString(policyAction, action)
-		fatalIf(probe.NewError(e), "Invalid pattern, please verify the pattern string.", nil)
+		matched, err := regexp.MatchString(policyAction, action)
+		fatalIf(err, "Invalid action \"%s\" in bucket policy.", action)
 		if matched {
 			return true
 		}
@@ -76,17 +75,45 @@ func bucketPolicyActionMatch(action string, statement policyStatement) bool {
 	return false
 }
 
+// Match function matches wild cards in 'pattern' for resource.
+func resourceMatch(pattern, resource string) bool {
+	if pattern == "" {
+		return resource == pattern
+	}
+	if pattern == "*" {
+		return true
+	}
+	parts := strings.Split(pattern, "*")
+	if len(parts) == 1 {
+		return resource == pattern
+	}
+	tGlob := strings.HasSuffix(pattern, "*")
+	end := len(parts) - 1
+	if !strings.HasPrefix(resource, parts[0]) {
+		return false
+	}
+	for i := 1; i < end; i++ {
+		if !strings.Contains(resource, parts[i]) {
+			return false
+		}
+		idx := strings.Index(resource, parts[i]) + len(parts[i])
+		resource = resource[idx:]
+	}
+	return tGlob || strings.HasSuffix(resource, parts[end])
+}
+
 // Verify if given resource matches with policy statement.
 func bucketPolicyResourceMatch(resource string, statement policyStatement) bool {
-	for _, presource := range statement.Resources {
-		matched, e := regexp.MatchString(presource, strings.TrimPrefix(resource, "/"))
-		fatalIf(probe.NewError(e), "Invalid pattern, please verify the pattern string.", nil)
-		// For any path matches, we return quickly and the let the caller continue.
-		if matched {
-			return true
+	for _, resourcep := range statement.Resources {
+		// the resource rule for object could contain "*" wild card.
+		// the requested object can be given access based on the already set bucket policy if
+		// the match is successful.
+		// More info: http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-arn-format.html .
+		if matched := resourceMatch(resourcep, resource); !matched {
+			return false
 		}
 	}
-	return false
+	return true
 }
 
 // Verify if given condition matches with policy statement.
@@ -127,7 +154,7 @@ func bucketPolicyConditionMatch(conditions map[string]string, statement policySt
 // -----------------
 // This implementation of the PUT operation uses the policy
 // subresource to add to or replace a policy on a bucket
-func (api objectStorageAPI) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -161,17 +188,17 @@ func (api objectStorageAPI) PutBucketPolicyHandler(w http.ResponseWriter, r *htt
 	// Read access policy up to maxAccessPolicySize.
 	// http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
 	// bucket policies are limited to 20KB in size, using a limit reader.
-	bucketPolicyBuf, e := ioutil.ReadAll(io.LimitReader(r.Body, maxAccessPolicySize))
-	if e != nil {
-		errorIf(probe.NewError(e).Trace(bucket), "Reading policy failed.", nil)
+	bucketPolicyBuf, err := ioutil.ReadAll(io.LimitReader(r.Body, maxAccessPolicySize))
+	if err != nil {
+		errorIf(err, "Unable to read bucket policy.")
 		writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
 		return
 	}
 
 	// Parse bucket policy.
-	bucketPolicy, e := parseBucketPolicy(bucketPolicyBuf)
-	if e != nil {
-		errorIf(probe.NewError(e), "Unable to parse bucket policy.", nil)
+	bucketPolicy, err := parseBucketPolicy(bucketPolicyBuf)
+	if err != nil {
+		errorIf(err, "Unable to parse bucket policy.")
 		writeErrorResponse(w, r, ErrInvalidPolicyDocument, r.URL.Path)
 		return
 	}
@@ -183,10 +210,9 @@ func (api objectStorageAPI) PutBucketPolicyHandler(w http.ResponseWriter, r *htt
 	}
 
 	// Save bucket policy.
-	err := writeBucketPolicy(bucket, bucketPolicyBuf)
-	if err != nil {
-		errorIf(err.Trace(bucket, string(bucketPolicyBuf)), "SaveBucketPolicy failed.", nil)
-		switch err.ToGoError().(type) {
+	if err := writeBucketPolicy(bucket, bucketPolicyBuf); err != nil {
+		errorIf(err, "Unable to write bucket policy.")
+		switch err.(type) {
 		case BucketNameInvalid:
 			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
 		default:
@@ -201,7 +227,7 @@ func (api objectStorageAPI) PutBucketPolicyHandler(w http.ResponseWriter, r *htt
 // -----------------
 // This implementation of the DELETE operation uses the policy
 // subresource to add to remove a policy on a bucket.
-func (api objectStorageAPI) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -218,10 +244,9 @@ func (api objectStorageAPI) DeleteBucketPolicyHandler(w http.ResponseWriter, r *
 	}
 
 	// Delete bucket access policy.
-	err := removeBucketPolicy(bucket)
-	if err != nil {
-		errorIf(err.Trace(bucket), "DeleteBucketPolicy failed.", nil)
-		switch err.ToGoError().(type) {
+	if err := removeBucketPolicy(bucket); err != nil {
+		errorIf(err, "Unable to remove bucket policy.")
+		switch err.(type) {
 		case BucketNameInvalid:
 			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
 		case BucketPolicyNotFound:
@@ -238,7 +263,7 @@ func (api objectStorageAPI) DeleteBucketPolicyHandler(w http.ResponseWriter, r *
 // -----------------
 // This operation uses the policy
 // subresource to return the policy of a specified bucket.
-func (api objectStorageAPI) GetBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) GetBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 
@@ -257,8 +282,8 @@ func (api objectStorageAPI) GetBucketPolicyHandler(w http.ResponseWriter, r *htt
 	// Read bucket access policy.
 	p, err := readBucketPolicy(bucket)
 	if err != nil {
-		errorIf(err.Trace(bucket), "GetBucketPolicy failed.", nil)
-		switch err.ToGoError().(type) {
+		errorIf(err, "Unable to read bucket policy.")
+		switch err.(type) {
 		case BucketNameInvalid:
 			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
 		case BucketPolicyNotFound:

bucket-policy-handlers_test.go

@@ -0,0 +1,78 @@
+/*
+ * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"fmt"
+	"testing"
+)
+
+// Tests validate Bucket policy resource matcher.
+func TestBucketPolicyResourceMatch(t *testing.T) {
+
+	// generates\ statement with given resource..
+	generateStatement := func(resource string) policyStatement {
+		statement := policyStatement{}
+		statement.Resources = []string{resource}
+		return statement
+	}
+
+	// generates resource prefix.
+	generateResource := func(bucketName, objectName string) string {
+		return AWSResourcePrefix + bucketName + "/" + objectName
+	}
+
+	testCases := []struct {
+		resourceToMatch       string
+		statement             policyStatement
+		expectedResourceMatch bool
+	}{
+		// Test case 1-4.
+		// Policy with resource ending with bucket/* allows access to all objects inside the given bucket.
+		{generateResource("minio-bucket", ""), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/*")), true},
+		{generateResource("minio-bucket", ""), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/*")), true},
+		{generateResource("minio-bucket", ""), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/*")), true},
+		{generateResource("minio-bucket", ""), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/*")), true},
+		// Test case - 5.
+		// Policy with resource ending with bucket/oo* should not allow access to bucket/output.txt.
+		{generateResource("minio-bucket", "output.txt"), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/oo*")), false},
+		// Test case - 6.
+		// Policy with resource ending with bucket/oo* should allow access to bucket/ootput.txt.
+		{generateResource("minio-bucket", "ootput.txt"), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/oo*")), true},
+		// Test case - 7.
+		// Policy with resource ending with bucket/oo* allows access to all subfolders starting with "oo" inside given bucket.
+		{generateResource("minio-bucket", "oop-bucket/my-file"), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/oo*")), true},
+		// Test case - 8.
+		{generateResource("minio-bucket", "Asia/India/1.pjg"), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/Asia/Japan/*")), false},
+		// Test case - 9.
+		{generateResource("minio-bucket", "Asia/India/1.pjg"), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix, "minio-bucket"+"/Asia/Japan/*")), false},
+		// Test case - 10.
+		// Proves that the name space is flat.
+		{generateResource("minio-bucket", "Africa/Bihar/India/design_info.doc/Bihar"), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix,
+			"minio-bucket"+"/*/India/*/Bihar")), true},
+		// Test case - 11.
+		// Proves that the name space is flat.
+		{generateResource("minio-bucket", "Asia/China/India/States/Bihar/output.txt"), generateStatement(fmt.Sprintf("%s%s", AWSResourcePrefix,
+			"minio-bucket"+"/*/India/*/Bihar/*")), true},
+	}
+	for i, testCase := range testCases {
+		actualResourceMatch := bucketPolicyResourceMatch(testCase.resourceToMatch, testCase.statement)
+		if testCase.expectedResourceMatch != actualResourceMatch {
+			t.Errorf("Test %d: Expected Resource match to be `%v`, but instead found it to be `%v`", i+1, testCase.expectedResourceMatch, actualResourceMatch)
+		}
+	}
+}

bucket-policy-parser.go

@@ -22,7 +22,7 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
-	"regexp"
+	"path"
 	"sort"
 	"strings"
 )
@@ -200,6 +200,14 @@ var invalidPrefixActions = map[string]struct{}{
 	// Add actions which do not honor prefixes.
 }
 
+// resourcePrefix - provides the prefix removing any wildcards.
+func resourcePrefix(resource string) string {
+	if strings.HasSuffix(resource, "*") {
+		resource = strings.TrimSuffix(resource, "*")
+	}
+	return path.Clean(resource)
+}
+
 // checkBucketPolicyResources validates Resources in unmarshalled bucket policy structure.
 // First valation of Resources done for given set of Actions.
 // Later its validated for recursive Resources.
@@ -232,7 +240,7 @@ func checkBucketPolicyResources(bucket string, bucketPolicy BucketPolicy) APIErr
 
 	var resources []string
 	for resource := range resourceMap {
-		resources = append(resources, resource)
+		resources = append(resources, resourcePrefix(resource))
 	}
 
 	// Sort strings as shorter first.
@@ -241,12 +249,12 @@ func checkBucketPolicyResources(bucket string, bucketPolicy BucketPolicy) APIErr
 	for len(resources) > 1 {
 		var resource string
 		resource, resources = resources[0], resources[1:]
-		resourceRegex := regexp.MustCompile(resource)
 		// Loop through all resources, if one of them matches with
 		// previous shorter one, it means we have detected
 		// nesting. Reject such rules.
 		for _, otherResource := range resources {
-			if resourceRegex.MatchString(otherResource) {
+			// Common prefix reject such rules.
+			if strings.HasPrefix(otherResource, resource) {
 				return ErrMalformedPolicy
 			}
 		}

bucket-policy-parser_test.go

@@ -453,31 +453,42 @@ func TestCheckBucketPolicyResources(t *testing.T) {
 		return statements
 	}
 
+	// constructing policy statement with lexically close characters.
+	// should not result in ErrMalformedPolicy
+	setResourceLexical := func(statements []policyStatement) []policyStatement {
+		statements[0].Resources = []string{"arn:aws:s3:::minio-bucket/op*", "arn:aws:s3:::minio-bucket/oo*"}
+		return statements
+	}
+
 	// List of BucketPolicy used for tests.
 	bucketAccessPolicies := []BucketPolicy{
-		// BucketPolicy - 0.
+		// BucketPolicy - 1.
 		// Contains valid read only policy statement.
 		{Version: "1.0", Statements: setReadOnlyStatement("minio-bucket", "")},
-		// BucketPolicy - 1.
+		// BucketPolicy - 2.
 		// Contains valid read-write only policy statement.
 		{Version: "1.0", Statements: setReadWriteStatement("minio-bucket", "Asia/")},
-		// BucketPolicy - 2.
+		// BucketPolicy - 3.
 		// Contains valid write only policy statement.
 		{Version: "1.0", Statements: setWriteOnlyStatement("minio-bucket", "Asia/India/")},
-		// BucketPolicy - 3.
+		// BucketPolicy - 4.
 		// Contains invalidPrefixActions.
 		// Since resourcePrefix is not to the bucket-name, it return ErrMalformedPolicy.
 		{Version: "1.0", Statements: setReadOnlyStatement("minio-bucket-fail", "Asia/India/")},
-		// BucketPolicy - 4.
+		// BucketPolicy - 5.
 		// constructing policy statement without invalidPrefixActions (check bucket-policy-parser.go).
 		// but bucket part of the resource is not equal to the bucket name.
 		// this results in return of ErrMalformedPolicy.
 		{Version: "1.0", Statements: setValidPrefixActions(setWriteOnlyStatement("minio-bucket-fail", "Asia/India/"))},
-		// BucketPolicy - 5.
-		// constructing policy statement without invalidPrefixActions (check bucket-policy-parser.go).
+		// BucketPolicy - 6.
 		// contructing policy statement with recursive resources.
 		// should result in ErrMalformedPolicy
 		{Version: "1.0", Statements: setRecurseResource(setValidPrefixActions(setWriteOnlyStatement("minio-bucket", "")))},
+		// BucketPolciy - 7.
+		// constructing policy statement with non recursive but
+		// lexically close resources.
+		// should result in ErrNone.
+		{Version: "1.0", Statements: setResourceLexical(setValidPrefixActions(setWriteOnlyStatement("minio-bucket", "oo")))},
 	}
 
 	testCases := []struct {
@@ -505,6 +516,11 @@ func TestCheckBucketPolicyResources(t *testing.T) {
 		// contructing policy statement with recursive resources.
 		// should result in ErrMalformedPolicy.
 		{bucketAccessPolicies[5], ErrMalformedPolicy, false},
+		// Test case - 7.
+		// constructing policy statement with lexically close
+		// characters.
+		// should result in ErrNone.
+		{bucketAccessPolicies[6], ErrNone, true},
 	}
 	for i, testCase := range testCases {
 		apiErrCode := checkBucketPolicyResources("minio-bucket", testCase.inputPolicy)

bucket-policy.go

@@ -20,132 +20,109 @@ import (
 	"io/ioutil"
 	"os"
 	"path/filepath"
-
-	"github.com/minio/minio/pkg/probe"
 )
 
 // getBucketsConfigPath - get buckets path.
-func getBucketsConfigPath() (string, *probe.Error) {
+func getBucketsConfigPath() (string, error) {
 	configPath, err := getConfigPath()
 	if err != nil {
-		return "", err.Trace()
+		return "", err
 	}
 	return filepath.Join(configPath, "buckets"), nil
 }
 
-// createBucketsConfigPath - create buckets directory.
-func createBucketsConfigPath() *probe.Error {
-	bucketsConfigPath, err := getBucketsConfigPath()
-	if err != nil {
-		return err
-	}
-	if e := os.MkdirAll(bucketsConfigPath, 0700); e != nil {
-		return probe.NewError(e)
-	}
-	return nil
-}
-
 // getBucketConfigPath - get bucket config path.
-func getBucketConfigPath(bucket string) (string, *probe.Error) {
+func getBucketConfigPath(bucket string) (string, error) {
 	bucketsConfigPath, err := getBucketsConfigPath()
 	if err != nil {
-		return "", err.Trace()
+		return "", err
 	}
 	return filepath.Join(bucketsConfigPath, bucket), nil
 }
 
 // createBucketConfigPath - create bucket config directory.
-func createBucketConfigPath(bucket string) *probe.Error {
+func createBucketConfigPath(bucket string) error {
 	bucketConfigPath, err := getBucketConfigPath(bucket)
 	if err != nil {
 		return err
 	}
-	if e := os.MkdirAll(bucketConfigPath, 0700); e != nil {
-		return probe.NewError(e)
-	}
-	return nil
+	return os.MkdirAll(bucketConfigPath, 0700)
 }
 
 // readBucketPolicy - read bucket policy.
-func readBucketPolicy(bucket string) ([]byte, *probe.Error) {
+func readBucketPolicy(bucket string) ([]byte, error) {
 	// Verify bucket is valid.
 	if !IsValidBucketName(bucket) {
-		return nil, probe.NewError(BucketNameInvalid{Bucket: bucket})
+		return nil, BucketNameInvalid{Bucket: bucket}
 	}
 
 	bucketConfigPath, err := getBucketConfigPath(bucket)
 	if err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
 
 	// Get policy file.
 	bucketPolicyFile := filepath.Join(bucketConfigPath, "access-policy.json")
-	if _, e := os.Stat(bucketPolicyFile); e != nil {
-		if os.IsNotExist(e) {
-			return nil, probe.NewError(BucketPolicyNotFound{Bucket: bucket})
+	if _, err = os.Stat(bucketPolicyFile); err != nil {
+		if os.IsNotExist(err) {
+			return nil, BucketPolicyNotFound{Bucket: bucket}
 		}
-		return nil, probe.NewError(e)
+		return nil, err
 	}
-
-	accessPolicyBytes, e := ioutil.ReadFile(bucketPolicyFile)
-	if e != nil {
-		return nil, probe.NewError(e)
-	}
-	return accessPolicyBytes, nil
+	return ioutil.ReadFile(bucketPolicyFile)
 }
 
 // removeBucketPolicy - remove bucket policy.
-func removeBucketPolicy(bucket string) *probe.Error {
+func removeBucketPolicy(bucket string) error {
 	// Verify bucket is valid.
 	if !IsValidBucketName(bucket) {
-		return probe.NewError(BucketNameInvalid{Bucket: bucket})
+		return BucketNameInvalid{Bucket: bucket}
 	}
 
 	bucketConfigPath, err := getBucketConfigPath(bucket)
 	if err != nil {
-		return err.Trace(bucket)
+		return err
 	}
 
 	// Get policy file.
 	bucketPolicyFile := filepath.Join(bucketConfigPath, "access-policy.json")
-	if _, e := os.Stat(bucketPolicyFile); e != nil {
-		if os.IsNotExist(e) {
-			return probe.NewError(BucketPolicyNotFound{Bucket: bucket})
+	if _, err = os.Stat(bucketPolicyFile); err != nil {
+		if os.IsNotExist(err) {
+			return BucketPolicyNotFound{Bucket: bucket}
 		}
-		return probe.NewError(e)
+		return err
+	}
+	if err := os.Remove(bucketPolicyFile); err != nil {
+		return err
 	}
 	return nil
 }
 
 // writeBucketPolicy - save bucket policy.
-func writeBucketPolicy(bucket string, accessPolicyBytes []byte) *probe.Error {
+func writeBucketPolicy(bucket string, accessPolicyBytes []byte) error {
 	// Verify if bucket path legal
 	if !IsValidBucketName(bucket) {
-		return probe.NewError(BucketNameInvalid{Bucket: bucket})
+		return BucketNameInvalid{Bucket: bucket}
 	}
 
 	// Create bucket config path.
 	if err := createBucketConfigPath(bucket); err != nil {
-		return err.Trace()
+		return err
 	}
 
 	bucketConfigPath, err := getBucketConfigPath(bucket)
 	if err != nil {
-		return err.Trace()
+		return err
 	}
 
 	// Get policy file.
 	bucketPolicyFile := filepath.Join(bucketConfigPath, "access-policy.json")
-	if _, e := os.Stat(bucketPolicyFile); e != nil {
-		if !os.IsNotExist(e) {
-			return probe.NewError(e)
+	if _, err := os.Stat(bucketPolicyFile); err != nil {
+		if !os.IsNotExist(err) {
+			return err
 		}
 	}
 
 	// Write bucket policy.
-	if e := ioutil.WriteFile(bucketPolicyFile, accessPolicyBytes, 0600); e != nil {
-		return probe.NewError(e)
-	}
-
-	return nil
+	return ioutil.WriteFile(bucketPolicyFile, accessPolicyBytes, 0600)
 }

build-constants.go

@@ -16,13 +16,18 @@
 
 package main
 
+// DO NOT EDIT THIS FILE DIRECTLY. These are build-time constants
+// set through ‘buildscripts/gen-ldflags.go’.
 var (
+	// minioGOPATH - GOPATH value at the time of build.
+	minioGOPATH = ""
+
 	// minioVersion - version time.RFC3339.
 	minioVersion = "DEVELOPMENT.GOGET"
 	// minioReleaseTag - release tag in TAG.%Y-%m-%dT%H-%M-%SZ.
 	minioReleaseTag = "DEVELOPMENT.GOGET"
 	// minioCommitID - latest commit id.
 	minioCommitID = "DEVELOPMENT.GOGET"
-	// minioShortCommitID - first 12 characters from mcCommitID
+	// minioShortCommitID - first 12 characters from minioCommitID.
 	minioShortCommitID = minioCommitID[:12]
 )

buildscripts/gen-ldflags.go

@@ -28,10 +28,11 @@ import (
 
 func genLDFlags(version string) string {
 	var ldflagsStr string
-	ldflagsStr = "-X main.minioVersion=" + version + " "
-	ldflagsStr = ldflagsStr + "-X main.minioReleaseTag=" + releaseTag(version) + " "
-	ldflagsStr = ldflagsStr + "-X main.minioCommitID=" + commitID() + " "
-	ldflagsStr = ldflagsStr + "-X main.minioShortCommitID=" + commitID()[:12]
+	ldflagsStr = "-X main.minioVersion=" + version
+	ldflagsStr += " -X main.minioReleaseTag=" + releaseTag(version)
+	ldflagsStr += " -X main.minioCommitID=" + commitID()
+	ldflagsStr += " -X main.minioShortCommitID=" + commitID()[:12]
+	ldflagsStr += " -X main.minioGOPATH=" + os.Getenv("GOPATH")
 	return ldflagsStr
 }
 

certs.go

@@ -21,26 +21,22 @@ import (
 	"path/filepath"
 
 	"github.com/minio/go-homedir"
-	"github.com/minio/minio/pkg/probe"
 )
 
 // createCertsPath create certs path.
-func createCertsPath() *probe.Error {
+func createCertsPath() error {
 	certsPath, err := getCertsPath()
 	if err != nil {
-		return err.Trace()
+		return err
 	}
-	if err := os.MkdirAll(certsPath, 0700); err != nil {
-		return probe.NewError(err)
-	}
-	return nil
+	return os.MkdirAll(certsPath, 0700)
 }
 
 // getCertsPath get certs path.
-func getCertsPath() (string, *probe.Error) {
-	homeDir, e := homedir.Dir()
-	if e != nil {
-		return "", probe.NewError(e)
+func getCertsPath() (string, error) {
+	homeDir, err := homedir.Dir()
+	if err != nil {
+		return "", err
 	}
 	certsPath := filepath.Join(homeDir, globalMinioCertsDir)
 	return certsPath, nil
@@ -49,7 +45,7 @@ func getCertsPath() (string, *probe.Error) {
 // mustGetCertsPath must get certs path.
 func mustGetCertsPath() string {
 	certsPath, err := getCertsPath()
-	fatalIf(err.Trace(), "Unable to retrieve certs path.", nil)
+	fatalIf(err, "Failed to get certificate path.")
 	return certsPath
 }
 

commands.go

@@ -26,19 +26,15 @@ var commandsTree = newTrie()
 
 // Collection of minio flags currently supported.
 var globalFlags = []cli.Flag{
-	cli.BoolFlag{
-		Name:  "quiet, q",
-		Usage: "Suppress chatty console output.",
-	},
-	cli.BoolFlag{
-		Name:  "debug",
-		Usage: "Enable debugging output.",
-	},
 	cli.StringFlag{
 		Name:  "config-dir, C",
 		Value: mustGetConfigPath(),
 		Usage: "Path to configuration folder.",
 	},
+	cli.BoolFlag{
+		Name:  "quiet",
+		Usage: "Suppress chatty output.",
+	},
 }
 
 // registerCommand registers a cli command.

config-migrate.go

@@ -22,7 +22,6 @@ import (
 	"path/filepath"
 
 	"github.com/minio/mc/pkg/console"
-	"github.com/minio/minio/pkg/probe"
 	"github.com/minio/minio/pkg/quick"
 )
 
@@ -38,35 +37,31 @@ func migrateConfig() {
 // Version '1' is not supported anymore and deprecated, safe to delete.
 func purgeV1() {
 	cv1, err := loadConfigV1()
-	if err != nil {
-		if os.IsNotExist(err.ToGoError()) {
-			return
-		}
+	if err != nil && os.IsNotExist(err) {
+		return
 	}
-	fatalIf(err.Trace(), "Unable to load config version ‘1’.", nil)
+	fatalIf(err, "Unable to load config version ‘1’.")
 
 	if cv1.Version == "1" {
-		console.Println("Unsupported config version ‘1’ found, removed successfully.")
+		console.Println("Removed unsupported config version ‘1’.")
 		/// Purge old fsUsers.json file
 		configPath, err := getConfigPath()
-		fatalIf(err.Trace(), "Unable to retrieve config path.", nil)
+		fatalIf(err, "Unable to retrieve config path.")
 
 		configFile := filepath.Join(configPath, "fsUsers.json")
 		os.RemoveAll(configFile)
 	}
-	fatalIf(probe.NewError(errors.New("")), "Unexpected version found ‘"+cv1.Version+"’, cannot migrate.", nil)
+	fatalIf(errors.New(""), "Failed to migrate unrecognized config version ‘"+cv1.Version+"’.")
 }
 
 // Version '2' to '3' config migration adds new fields and re-orders
 // previous fields. Simplifies config for future additions.
 func migrateV2ToV3() {
 	cv2, err := loadConfigV2()
-	if err != nil {
-		if os.IsNotExist(err.ToGoError()) {
-			return
-		}
+	if err != nil && os.IsNotExist(err) {
+		return
 	}
-	fatalIf(err.Trace(), "Unable to load config version ‘2’.", nil)
+	fatalIf(err, "Unable to load config version ‘2’.")
 	if cv2.Version != "2" {
 		return
 	}
@@ -78,6 +73,10 @@ func migrateV2ToV3() {
 		SecretAccessKey: cv2.Credentials.SecretAccessKey,
 	}
 	srvConfig.Region = cv2.Credentials.Region
+	if srvConfig.Region == "" {
+		// Region needs to be set for AWS Signature V4.
+		srvConfig.Region = "us-east-1"
+	}
 	srvConfig.Logger.Console = consoleLogger{
 		Enable: true,
 		Level:  "fatal",
@@ -99,14 +98,14 @@ func migrateV2ToV3() {
 	srvConfig.Logger.Syslog = slogger
 
 	qc, err := quick.New(srvConfig)
-	fatalIf(err.Trace(), "Unable to initialize config.", nil)
+	fatalIf(err, "Unable to initialize config.")
 
 	configFile, err := getConfigFile()
-	fatalIf(err.Trace(), "Unable to get config file.", nil)
+	fatalIf(err, "Unable to get config file.")
 
 	// Migrate the config.
 	err = qc.Save(configFile)
-	fatalIf(err.Trace(), "Migrating from version ‘"+cv2.Version+"’ to ‘"+srvConfig.Version+"’ failed.", nil)
+	fatalIf(err, "Failed to migrate config from ‘"+cv2.Version+"’ to ‘"+srvConfig.Version+"’ failed.")
 
 	console.Println("Migration from version ‘" + cv2.Version + "’ to ‘" + srvConfig.Version + "’ completed successfully.")
 }
@@ -116,12 +115,10 @@ func migrateV2ToV3() {
 // the config for future additions.
 func migrateV3ToV4() {
 	cv3, err := loadConfigV3()
-	if err != nil {
-		if os.IsNotExist(err.ToGoError()) {
-			return
-		}
+	if err != nil && os.IsNotExist(err) {
+		return
 	}
-	fatalIf(err.Trace(), "Unable to load config version ‘3’.", nil)
+	fatalIf(err, "Unable to load config version ‘3’.")
 	if cv3.Version != "3" {
 		return
 	}
@@ -131,17 +128,21 @@ func migrateV3ToV4() {
 	srvConfig.Version = globalMinioConfigVersion
 	srvConfig.Credential = cv3.Credential
 	srvConfig.Region = cv3.Region
+	if srvConfig.Region == "" {
+		// Region needs to be set for AWS Signature Version 4.
+		srvConfig.Region = "us-east-1"
+	}
 	srvConfig.Logger.Console = cv3.Logger.Console
 	srvConfig.Logger.File = cv3.Logger.File
 	srvConfig.Logger.Syslog = cv3.Logger.Syslog
 
 	qc, err := quick.New(srvConfig)
-	fatalIf(err.Trace(), "Unable to initialize the quick config.", nil)
+	fatalIf(err, "Unable to initialize the quick config.")
 	configFile, err := getConfigFile()
-	fatalIf(err.Trace(), "Unable to get config file.", nil)
+	fatalIf(err, "Unable to get config file.")
 
 	err = qc.Save(configFile)
-	fatalIf(err.Trace(), "Migrating from version ‘"+cv3.Version+"’ to ‘"+srvConfig.Version+"’ failed.", nil)
+	fatalIf(err, "Failed to migrate config from ‘"+cv3.Version+"’ to ‘"+srvConfig.Version+"’ failed.")
 
 	console.Println("Migration from version ‘" + cv3.Version + "’ to ‘" + srvConfig.Version + "’ completed successfully.")
 }

config-old.go

@@ -4,7 +4,6 @@ import (
 	"os"
 	"path/filepath"
 
-	"github.com/minio/minio/pkg/probe"
 	"github.com/minio/minio/pkg/quick"
 )
 
@@ -16,25 +15,25 @@ type configV1 struct {
 }
 
 // loadConfigV1 load config
-func loadConfigV1() (*configV1, *probe.Error) {
+func loadConfigV1() (*configV1, error) {
 	configPath, err := getConfigPath()
 	if err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
 	configFile := filepath.Join(configPath, "fsUsers.json")
-	if _, err := os.Stat(configFile); err != nil {
-		return nil, probe.NewError(err)
+	if _, err = os.Stat(configFile); err != nil {
+		return nil, err
 	}
-	a := &configV1{}
-	a.Version = "1"
-	qc, err := quick.New(a)
+	c := &configV1{}
+	c.Version = "1"
+	qc, err := quick.New(c)
 	if err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
 	if err := qc.Load(configFile); err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
-	return qc.Data().(*configV1), nil
+	return c, nil
 }
 
 /////////////////// Config V2 ///////////////////
@@ -60,24 +59,24 @@ type configV2 struct {
 }
 
 // loadConfigV2 load config version '2'.
-func loadConfigV2() (*configV2, *probe.Error) {
+func loadConfigV2() (*configV2, error) {
 	configFile, err := getConfigFile()
 	if err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
-	if _, err := os.Stat(configFile); err != nil {
-		return nil, probe.NewError(err)
+	if _, err = os.Stat(configFile); err != nil {
+		return nil, err
 	}
-	a := &configV2{}
-	a.Version = "2"
-	qc, err := quick.New(a)
+	c := &configV2{}
+	c.Version = "2"
+	qc, err := quick.New(c)
 	if err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
 	if err := qc.Load(configFile); err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
-	return qc.Data().(*configV2), nil
+	return c, nil
 }
 
 /////////////////// Config V3 ///////////////////
@@ -127,22 +126,22 @@ type configV3 struct {
 }
 
 // loadConfigV3 load config version '3'.
-func loadConfigV3() (*configV3, *probe.Error) {
+func loadConfigV3() (*configV3, error) {
 	configFile, err := getConfigFile()
 	if err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
-	if _, err := os.Stat(configFile); err != nil {
-		return nil, probe.NewError(err)
+	if _, err = os.Stat(configFile); err != nil {
+		return nil, err
 	}
-	a := &configV3{}
-	a.Version = "3"
-	qc, err := quick.New(a)
+	c := &configV3{}
+	c.Version = "3"
+	qc, err := quick.New(c)
 	if err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
 	if err := qc.Load(configFile); err != nil {
-		return nil, err.Trace()
+		return nil, err
 	}
-	return qc.Data().(*configV3), nil
+	return c, nil
 }

config-v4.go

@@ -20,7 +20,6 @@ import (
 	"os"
 	"sync"
 
-	"github.com/minio/minio/pkg/probe"
 	"github.com/minio/minio/pkg/quick"
 )
 
@@ -40,7 +39,7 @@ type serverConfigV4 struct {
 }
 
 // initConfig - initialize server config. config version (called only once).
-func initConfig() *probe.Error {
+func initConfig() error {
 	if !isConfigFileExists() {
 		srvCfg := &serverConfigV4{}
 		srvCfg.Version = globalMinioConfigVersion
@@ -55,44 +54,40 @@ func initConfig() *probe.Error {
 		// Create config path.
 		err := createConfigPath()
 		if err != nil {
-			return err.Trace()
+			return err
 		}
 
 		// Create certs path.
 		err = createCertsPath()
 		if err != nil {
-			return err.Trace()
+			return err
 		}
 
 		// Save the new config globally.
 		serverConfig = srvCfg
 
 		// Save config into file.
-		err = serverConfig.Save()
-		if err != nil {
-			return err.Trace()
-		}
-		return nil
+		return serverConfig.Save()
 	}
 	configFile, err := getConfigFile()
 	if err != nil {
-		return err.Trace()
+		return err
 	}
-	if _, e := os.Stat(configFile); err != nil {
-		return probe.NewError(e)
+	if _, err = os.Stat(configFile); err != nil {
+		return err
 	}
 	srvCfg := &serverConfigV4{}
 	srvCfg.Version = globalMinioConfigVersion
 	srvCfg.rwMutex = &sync.RWMutex{}
 	qc, err := quick.New(srvCfg)
 	if err != nil {
-		return err.Trace()
+		return err
 	}
 	if err := qc.Load(configFile); err != nil {
-		return err.Trace()
+		return err
 	}
 	// Save the loaded config globally.
-	serverConfig = qc.Data().(*serverConfigV4)
+	serverConfig = srvCfg
 	// Set the version properly after the unmarshalled json is loaded.
 	serverConfig.Version = globalMinioConfigVersion
 	return nil
@@ -181,27 +176,22 @@ func (s serverConfigV4) GetCredential() credential {
 }
 
 // Save config.
-func (s serverConfigV4) Save() *probe.Error {
+func (s serverConfigV4) Save() error {
 	s.rwMutex.RLock()
 	defer s.rwMutex.RUnlock()
 
 	// get config file.
 	configFile, err := getConfigFile()
 	if err != nil {
-		return err.Trace()
+		return err
 	}
 
 	// initialize quick.
 	qc, err := quick.New(&s)
 	if err != nil {
-		return err.Trace()
+		return err
 	}
 
 	// Save config file.
-	if err := qc.Save(configFile); err != nil {
-		return err.Trace()
-	}
-
-	// Return success.
-	return nil
+	return qc.Save(configFile)
 }

config.go

@@ -21,7 +21,6 @@ import (
 	"path/filepath"
 
 	"github.com/minio/go-homedir"
-	"github.com/minio/minio/pkg/probe"
 )
 
 // configPath for custom config path only for testing purposes
@@ -33,13 +32,13 @@ func setGlobalConfigPath(configPath string) {
 }
 
 // getConfigPath get server config path
-func getConfigPath() (string, *probe.Error) {
+func getConfigPath() (string, error) {
 	if customConfigPath != "" {
 		return customConfigPath, nil
 	}
-	homeDir, e := homedir.Dir()
-	if e != nil {
-		return "", probe.NewError(e)
+	homeDir, err := homedir.Dir()
+	if err != nil {
+		return "", err
 	}
 	configPath := filepath.Join(homeDir, globalMinioConfigDir)
 	return configPath, nil
@@ -48,27 +47,24 @@ func getConfigPath() (string, *probe.Error) {
 // mustGetConfigPath must get server config path.
 func mustGetConfigPath() string {
 	configPath, err := getConfigPath()
-	fatalIf(err.Trace(), "Unable to get config path.", nil)
+	fatalIf(err, "Unable to get config path.")
 	return configPath
 }
 
 // createConfigPath create server config path.
-func createConfigPath() *probe.Error {
+func createConfigPath() error {
 	configPath, err := getConfigPath()
 	if err != nil {
-		return err.Trace()
+		return err
 	}
-	if err := os.MkdirAll(configPath, 0700); err != nil {
-		return probe.NewError(err)
-	}
-	return nil
+	return os.MkdirAll(configPath, 0700)
 }
 
 // isConfigFileExists - returns true if config file exists.
 func isConfigFileExists() bool {
-	st, e := os.Stat(mustGetConfigFile())
+	st, err := os.Stat(mustGetConfigFile())
 	// If file exists and is regular return true.
-	if e == nil && st.Mode().IsRegular() {
+	if err == nil && st.Mode().IsRegular() {
 		return true
 	}
 	return false
@@ -77,16 +73,16 @@ func isConfigFileExists() bool {
 // mustGetConfigFile must get server config file.
 func mustGetConfigFile() string {
 	configFile, err := getConfigFile()
-	fatalIf(err.Trace(), "Unable to get config file.", nil)
+	fatalIf(err, "Unable to get config file.")
 
 	return configFile
 }
 
 // getConfigFile get server config file.
-func getConfigFile() (string, *probe.Error) {
+func getConfigFile() (string, error) {
 	configPath, err := getConfigPath()
 	if err != nil {
-		return "", err.Trace()
+		return "", err
 	}
 	return filepath.Join(configPath, globalMinioConfigFile), nil
 }

docker/Dockerfile

@@ -5,4 +5,6 @@ ADD minio.dockerimage /minio
 ADD export /export
 EXPOSE 9000
 COPY start.sh /start.sh
+ENV ALLOW_CONTAINER_ROOT=1
 ENTRYPOINT ["/start.sh"]
+CMD ["/export"]

docs/backend-multipart.md

@@ -0,0 +1,33 @@
+## Multipart
+
+Each incoming part is uploaded in the following format.
+
+Placeholder uploads.json to indicate a leaf.
+
+ ```EXPORT_DIR/.minio/multipart/BUCKET/PATH/TO/OBJECT/uploads.json```
+
+Incomplete file
+
+ ```EXPORT_DIR/.minio/multipart/BUCKET/PATH/TO/OBJECT/UPLOADID/00000.incomplete```
+
+Actual parts
+
+ ```EXPORT_DIR/.minio/multipart/BUCKET/PATH/TO/OBJECT/UPLOADID/PART_NUMBER.MD5SUM_STRING```
+
+## FS Format.
+
+Each of these parts are concatenated back to a single contigous file.
+
+```EXPORT_DIR/BUCKET/PATH/TO/OBJECT```
+
+## XL Format.
+
+Each of these parts are kept as is in the following format.
+
+Special json file indicate the metata multipart information, essentially list of parts etc.
+
+```EXPORT_DIR/BUCKET/PATH/TO/OBJECT/00000.minio.multipart```
+
+All the parts that were uploaded.
+
+```EXPORT_DIR/BUCKET/PATH/TO/OBJECT/PART_NUMBER.minio.multipart```

docs/backend/fs/format.json

@@ -0,0 +1,4 @@
+{
+  "format": "fs",
+  "version": "1"
+}

docs/backend/fs/fs.json

@@ -0,0 +1,15 @@
+{
+    "version": "1",
+    "format": "fs",
+    "minio": {
+        "release": "DEVELOPMENT.GOGET"
+    },
+    "parts": [
+        {
+            "number": 1,
+            "name": "object1",
+            "size": 29,
+            "eTag": "",
+        },
+    ]
+}

docs/backend/fs/uploads.json

@@ -0,0 +1,10 @@
+{
+    "version": "1",
+    "format": "fs",
+    "uploadIds": [
+        {
+            "uploadID": "id",
+            "startTime": "time",
+        }
+    ]
+}

docs/backend/xl/format.json

@@ -0,0 +1,20 @@
+{
+  "xl": {
+    "jbod": [
+      "8aa2b1bc-0e5a-49e0-8221-05228336b040",
+      "3467a69b-0266-478a-9e10-e819447e4545",
+      "d4a4505b-4e4f-4864-befd-4f36adb0bc66",
+      "592b6583-ca26-47af-b991-ba6d097e34e8",
+      "c7ef69f0-dbf5-4c0e-b167-d30a441bad7e",
+      "f0b36ea3-fe96-4f2b-bced-22c7f33e0e0c",
+      "b83abf39-e39d-4e7b-8e16-6f9953455a48",
+      "7d63dfc9-5441-4243-bd36-de8db0691982",
+      "c1bbffc5-81f9-4251-9398-33a959b3ce37",
+      "64408f94-26e0-4277-9593-2d703f4d5a91"
+    ],
+    "disk": "8aa2b1bc-0e5a-49e0-8221-05228336b040",
+    "version": "1"
+  },
+  "format": "xl",
+  "version": "1"
+}

docs/backend/xl/uploads.json

@@ -0,0 +1,10 @@
+{
+    "version": "1",
+    "format": "xl",
+    "uploadIds": [
+        {
+            "uploadID": "id",
+            "startTime": "time",
+        }
+    ]
+}

docs/backend/xl/xl.json

@@ -0,0 +1,57 @@
+{
+    "parts": [
+        {
+            "number": 1,
+            "size": 5242880,
+            "etag": "3565c6e741e69a007a5ac7db893a62b5",
+            "name": "object1"
+        },
+        {
+            "number": 2,
+            "size": 5242880,
+            "etag": "d416712335c280ab1e39498552937764",
+            "name": "object2"
+        },
+        {
+            "number": 3,
+            "size": 4338324,
+            "etag": "8a98c5c54d81c6c95ed9bdcaeb941aaf",
+            "name": "object3"
+        }
+    ],
+    "meta": {
+        "md5Sum": "97586a5290d4f5a41328062d6a7da593-3",
+        "content-type": "application\/octet-stream",
+        "content-encoding": ""
+    },
+    "minio": {
+        "release": "DEVELOPMENT.GOGET"
+    },
+    "erasure": {
+        "algorithm": "klauspost/reedsolomon/vandermonde",
+        "index": 2,
+        "distribution": [ 1, 3, 4, 2, 5, 8, 7, 6, 9 ],
+        "blockSize": 4194304,
+        "parity": 5,
+        "data": 5,
+        "checksum": [
+            {
+                "name": "object1",
+                "algorithm": "sha512",
+                "hash": "d9910e1492446389cfae6fe979db0245f96ca97ca2c7a25cab45805882004479320d866a47ea1f7be6a62625dd4de6caf7816009ef9d62779346d01a221b335c",
+            },
+            {
+                "name": "object2",
+                "algorithm": "sha512",
+                "hash": "d9910e1492446389cfae6fe979db0245f96ca97ca2c7a25cab45805882004479320d866a47ea1f7be6a62625dd4de6caf7816009ef9d62779346d01a221b335c",
+            },
+        ],
+    },
+    "stat": {
+        "version": 0,
+        "modTime": "2016-05-24T00:09:40.122390255Z",
+        "size": 14824084
+    },
+    "format": "xl",
+    "version": "1"
+}

docs/logging.md

@@ -0,0 +1,28 @@
+### Logging.
+
+- `fatalIf` - wrapper function which takes error and prints jsonic error messages.
+- `errorIf` - similar to fatalIf but doesn't exit on err != nil.
+
+Supported logging targets.
+
+- console
+- file
+- syslog
+
+Sample logger section from `~/.minio/config.json`
+```
+		"console": {
+			"enable": true,
+			"level": "error"
+		},
+		"file": {
+			"enable": false,
+			"fileName": "",
+			"level": "error"
+		},
+		"syslog": {
+			"enable": false,
+			"address": "",
+			"level": "error"
+		}
+```

docs/web-browser.md

@@ -19,21 +19,22 @@ minio server <testdir>
 
 JSON RPC namespace is `Web`.
 
-#### Auth Operations
+#### Server operations.
+
+* ServerInfo - fetches current server information, includes memory statistics, minio binary
+  version, golang runtime version and more.
+
+#### Auth operations
 
 * Login - waits for 'username, password' and on success replies a new Json Web Token (JWT).
-* ResetToken - resets token, requires password and token.
-* Logout - currently a dummy operation.
+* SetAuth - change access credentials with new 'username, password'.
+* GetAuth - fetch the current auth from the server.
 
-#### Bucket/Object Operations.
+#### Bucket/Object operations.
 
 * ListBuckets - lists buckets, requires a valid token.
 * ListObjects - lists objects, requires a valid token.
 * MakeBucket - make a new bucket, requires a valid token.
-* GetObjectURL - generates a URL for download access, requires a valid token.
-  (generated URL is valid for 1hr)
-* PutObjectURL - generates a URL for upload access, requies a valid token.
-  (generated URL is valid for 1hr)
-
-#### Server Operations. 
-* DiskInfo - get backend disk statistics.
+* RemoveObject - removes an object from a bucket, requires a valid token.
+* Upload - uploads a new object from the browser, requires a valid token.
+* Download - downloads an object from a bucket, requires a valid token.

docs/xl-file-json.md

@@ -0,0 +1,41 @@
+### file.json
+
+``file.json`` is a special file captured and written by XL storage API layer
+to interpret, manage and extract erasured data from multiple disks.
+
+```json
+{
+    "version": "1.0.0",
+    "stat": {
+        "size": 24256,
+        "modTime": "2016-04-28T00:11:37.843Z",
+        "delete": false,
+        "version": 0
+    },
+    "erasure": {
+        "data": 5,
+        "parity": 5,
+        "blockSize": 4194304
+    ],
+    "minio": {
+        "release": "RELEASE.2016-04-28T00-09-47Z"
+    }
+}
+```
+
+#### JSON meaning.
+
+- "version" // Version of the meta json file.
+
+- "stat" // Stat value of written file.
+
+  - "size"     // Size of the file.
+  - "modTime"  // Modified time of the file.
+  - "deleted"  // Field to track if the file is deleted when disks are down.
+  - "version"  // File version tracked when disks are down.
+
+- "erasure" // Erasure metadata for the written file.
+
+  - "data"       // Data blocks parts of the file.
+  - "parity"     // Parity blocks parts of the file.
+  - "blockSize"  // BlockSize read/write chunk size.

erasure-createfile.go

@@ -0,0 +1,149 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"encoding/hex"
+	"hash"
+	"io"
+	"sync"
+
+	"github.com/klauspost/reedsolomon"
+)
+
+// erasureCreateFile - writes an entire stream by erasure coding to
+// all the disks, writes also calculate individual block's checksum
+// for future bit-rot protection.
+func erasureCreateFile(disks []StorageAPI, volume string, path string, partName string, data io.Reader, eInfos []erasureInfo) (newEInfos []erasureInfo, size int64, err error) {
+	// Allocated blockSized buffer for reading.
+	buf := make([]byte, blockSizeV1)
+	hashWriters := newHashWriters(len(disks))
+
+	// Just pick one eInfo.
+	eInfo := pickValidErasureInfo(eInfos)
+
+	// Read until io.EOF, erasure codes data and writes to all disks.
+	for {
+		var n int
+		n, err = io.ReadFull(data, buf)
+		if err == io.EOF {
+			break
+		}
+		if err != nil && err != io.ErrUnexpectedEOF {
+			return nil, 0, err
+		}
+		size += int64(n)
+		var blocks [][]byte
+		// Returns encoded blocks.
+		blocks, err = encodeData(buf[:n], eInfo.DataBlocks, eInfo.ParityBlocks)
+		if err != nil {
+			return nil, 0, err
+		}
+		err = appendFile(disks, volume, path, blocks, eInfo.Distribution, hashWriters)
+		if err != nil {
+			return nil, 0, err
+		}
+	}
+
+	// Save the checksums.
+	checkSums := make([]checkSumInfo, len(disks))
+	for index := range disks {
+		blockIndex := eInfo.Distribution[index] - 1
+		checkSums[blockIndex] = checkSumInfo{
+			Name:      partName,
+			Algorithm: "sha512",
+			Hash:      hex.EncodeToString(hashWriters[blockIndex].Sum(nil)),
+		}
+	}
+
+	// Erasure info update for checksum for each disks.
+	newEInfos = make([]erasureInfo, len(disks))
+	for index, eInfo := range eInfos {
+		if eInfo.IsValid() {
+			blockIndex := eInfo.Distribution[index] - 1
+			newEInfos[index] = eInfo
+			newEInfos[index].Checksum = append(newEInfos[index].Checksum, checkSums[blockIndex])
+		}
+	}
+
+	// Return newEInfos.
+	return newEInfos, size, nil
+}
+
+// encodeData - encodes incoming data buffer into
+// dataBlocks+parityBlocks returns a 2 dimensional byte array.
+func encodeData(dataBuffer []byte, dataBlocks, parityBlocks int) ([][]byte, error) {
+	rs, err := reedsolomon.New(dataBlocks, parityBlocks)
+	if err != nil {
+		return nil, err
+	}
+	// Split the input buffer into data and parity blocks.
+	var blocks [][]byte
+	blocks, err = rs.Split(dataBuffer)
+	if err != nil {
+		return nil, err
+	}
+
+	// Encode parity blocks using data blocks.
+	err = rs.Encode(blocks)
+	if err != nil {
+		return nil, err
+	}
+
+	// Return encoded blocks.
+	return blocks, nil
+}
+
+// appendFile - append data buffer at path.
+func appendFile(disks []StorageAPI, volume, path string, enBlocks [][]byte, distribution []int, hashWriters []hash.Hash) (err error) {
+	var wg = &sync.WaitGroup{}
+	var wErrs = make([]error, len(disks))
+	// Write encoded data to quorum disks in parallel.
+	for index, disk := range disks {
+		if disk == nil {
+			continue
+		}
+		wg.Add(1)
+		// Write encoded data in routine.
+		go func(index int, disk StorageAPI) {
+			defer wg.Done()
+			// Pick the block from the distribution.
+			blockIndex := distribution[index] - 1
+			n, wErr := disk.AppendFile(volume, path, enBlocks[blockIndex])
+			if wErr != nil {
+				wErrs[index] = wErr
+				return
+			}
+			if n != int64(len(enBlocks[blockIndex])) {
+				wErrs[index] = errUnexpected
+				return
+			}
+
+			// Calculate hash for each blocks.
+			hashWriters[blockIndex].Write(enBlocks[blockIndex])
+
+			// Successfully wrote.
+			wErrs[index] = nil
+		}(index, disk)
+	}
+
+	// Wait for all the appends to finish.
+	wg.Wait()
+
+	// Return success.
+	return nil
+}

erasure-readfile.go

@@ -0,0 +1,208 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"encoding/hex"
+	"errors"
+
+	"github.com/klauspost/reedsolomon"
+)
+
+// erasureReadFile - read an entire erasure coded file at into a byte
+// array. Erasure coded parts are often few mega bytes in size and it
+// is convenient to return them as byte slice. This function also
+// supports bit-rot detection by verifying checksum of individual
+// block's checksum.
+func erasureReadFile(disks []StorageAPI, volume string, path string, partName string, size int64, eInfos []erasureInfo) ([]byte, error) {
+	// Return data buffer.
+	var buffer []byte
+
+	// Total size left
+	totalSizeLeft := size
+
+	// Starting offset for reading.
+	startOffset := int64(0)
+
+	// Gather previously calculated block checksums.
+	blockCheckSums := metaPartBlockChecksums(disks, eInfos, partName)
+
+	// Pick one erasure info.
+	eInfo := pickValidErasureInfo(eInfos)
+
+	// Write until each parts are read and exhausted.
+	for totalSizeLeft > 0 {
+		// Calculate the proper block size.
+		var curBlockSize int64
+		if eInfo.BlockSize < totalSizeLeft {
+			curBlockSize = eInfo.BlockSize
+		} else {
+			curBlockSize = totalSizeLeft
+		}
+
+		// Calculate the current encoded block size.
+		curEncBlockSize := getEncodedBlockLen(curBlockSize, eInfo.DataBlocks)
+		offsetEncOffset := getEncodedBlockLen(startOffset, eInfo.DataBlocks)
+
+		// Allocate encoded blocks up to storage disks.
+		enBlocks := make([][]byte, len(disks))
+
+		// Counter to keep success data blocks.
+		var successDataBlocksCount = 0
+		var noReconstruct bool // Set for no reconstruction.
+
+		// Read from all the disks.
+		for index, disk := range disks {
+			blockIndex := eInfo.Distribution[index] - 1
+			if !isValidBlock(disks, volume, path, toDiskIndex(blockIndex, eInfo.Distribution), blockCheckSums) {
+				continue
+			}
+			if disk == nil {
+				continue
+			}
+			// Initialize shard slice and fill the data from each parts.
+			enBlocks[blockIndex] = make([]byte, curEncBlockSize)
+			// Read the necessary blocks.
+			_, err := disk.ReadFile(volume, path, offsetEncOffset, enBlocks[blockIndex])
+			if err != nil {
+				enBlocks[blockIndex] = nil
+			}
+			// Verify if we have successfully read all the data blocks.
+			if blockIndex < eInfo.DataBlocks && enBlocks[blockIndex] != nil {
+				successDataBlocksCount++
+				// Set when we have all the data blocks and no
+				// reconstruction is needed, so that we can avoid
+				// erasure reconstruction.
+				noReconstruct = successDataBlocksCount == eInfo.DataBlocks
+				if noReconstruct {
+					// Break out we have read all the data blocks.
+					break
+				}
+			}
+		}
+
+		// Check blocks if they are all zero in length, we have corruption return error.
+		if checkBlockSize(enBlocks) == 0 {
+			return nil, errXLDataCorrupt
+		}
+
+		// Verify if reconstruction is needed, proceed with reconstruction.
+		if !noReconstruct {
+			err := decodeData(enBlocks, eInfo.DataBlocks, eInfo.ParityBlocks)
+			if err != nil {
+				return nil, err
+			}
+		}
+
+		// Get data blocks from encoded blocks.
+		dataBlocks, err := getDataBlocks(enBlocks, eInfo.DataBlocks, int(curBlockSize))
+		if err != nil {
+			return nil, err
+		}
+
+		// Copy data blocks.
+		buffer = append(buffer, dataBlocks...)
+
+		// Negate the 'n' size written to client.
+		totalSizeLeft -= int64(len(dataBlocks))
+
+		// Increase the offset to move forward.
+		startOffset += int64(len(dataBlocks))
+
+		// Relenquish memory.
+		dataBlocks = nil
+	}
+	return buffer, nil
+}
+
+// PartObjectChecksum - returns the checksum for the part name from the checksum slice.
+func (e erasureInfo) PartObjectChecksum(partName string) checkSumInfo {
+	for _, checksum := range e.Checksum {
+		if checksum.Name == partName {
+			return checksum
+		}
+	}
+	return checkSumInfo{}
+}
+
+// xlMetaPartBlockChecksums - get block checksums for a given part.
+func metaPartBlockChecksums(disks []StorageAPI, eInfos []erasureInfo, partName string) (blockCheckSums []checkSumInfo) {
+	for index := range disks {
+		if eInfos[index].IsValid() {
+			// Save the read checksums for a given part.
+			blockCheckSums = append(blockCheckSums, eInfos[index].PartObjectChecksum(partName))
+		} else {
+			blockCheckSums = append(blockCheckSums, checkSumInfo{})
+		}
+	}
+	return blockCheckSums
+}
+
+// Takes block index and block distribution to get the disk index.
+func toDiskIndex(blockIdx int, distribution []int) (diskIndex int) {
+	diskIndex = -1
+	// Find out the right disk index for the input block index.
+	for index, blockIndex := range distribution {
+		if blockIndex == blockIdx {
+			diskIndex = index
+		}
+	}
+	return diskIndex
+}
+
+// isValidBlock - calculates the checksum hash for the block and
+// validates if its correct returns true for valid cases, false otherwise.
+func isValidBlock(disks []StorageAPI, volume, path string, diskIndex int, blockCheckSums []checkSumInfo) bool {
+	// Unknown block index requested, treat it as error.
+	if diskIndex == -1 {
+		return false
+	}
+	// Disk is not present, treat entire block to be non existent.
+	if disks[diskIndex] == nil {
+		return false
+	}
+	// Read everything for a given block and calculate hash.
+	hashWriter := newHash(blockCheckSums[diskIndex].Algorithm)
+	hashBytes, err := hashSum(disks[diskIndex], volume, path, hashWriter)
+	if err != nil {
+		return false
+	}
+	return hex.EncodeToString(hashBytes) == blockCheckSums[diskIndex].Hash
+}
+
+// decodeData - decode encoded blocks.
+func decodeData(enBlocks [][]byte, dataBlocks, parityBlocks int) error {
+	rs, err := reedsolomon.New(dataBlocks, parityBlocks)
+	if err != nil {
+		return err
+	}
+	err = rs.Reconstruct(enBlocks)
+	if err != nil {
+		return err
+	}
+	// Verify reconstructed blocks (parity).
+	ok, err := rs.Verify(enBlocks)
+	if err != nil {
+		return err
+	}
+	if !ok {
+		// Blocks cannot be reconstructed, corrupted data.
+		err = errors.New("Verification failed after reconstruction, data likely corrupted.")
+		return err
+	}
+	return nil
+}

erasure-utils.go

@@ -0,0 +1,108 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"crypto/sha512"
+	"hash"
+	"io"
+
+	"github.com/klauspost/reedsolomon"
+)
+
+// newHashWriters - inititialize a slice of hashes for the disk count.
+func newHashWriters(diskCount int) []hash.Hash {
+	hashWriters := make([]hash.Hash, diskCount)
+	for index := range hashWriters {
+		hashWriters[index] = newHash("sha512")
+	}
+	return hashWriters
+}
+
+// newHash - gives you a newly allocated hash depending on the input algorithm.
+func newHash(algo string) hash.Hash {
+	switch algo {
+	case "sha512":
+		return sha512.New()
+	// Add new hashes here.
+	default:
+		return sha512.New()
+	}
+}
+
+func hashSum(disk StorageAPI, volume, path string, writer hash.Hash) ([]byte, error) {
+	startOffset := int64(0)
+	// Read until io.EOF.
+	for {
+		buf := make([]byte, blockSizeV1)
+		n, err := disk.ReadFile(volume, path, startOffset, buf)
+		if err == io.EOF {
+			break
+		}
+		if err != nil && err != io.EOF {
+			return nil, err
+		}
+		writer.Write(buf[:n])
+		startOffset += n
+	}
+	return writer.Sum(nil), nil
+}
+
+// getDataBlocks - fetches the data block only part of the input encoded blocks.
+func getDataBlocks(enBlocks [][]byte, dataBlocks int, curBlockSize int) (data []byte, err error) {
+	if len(enBlocks) < dataBlocks {
+		return nil, reedsolomon.ErrTooFewShards
+	}
+	size := 0
+	blocks := enBlocks[:dataBlocks]
+	for _, block := range blocks {
+		size += len(block)
+	}
+	if size < curBlockSize {
+		return nil, reedsolomon.ErrShortData
+	}
+
+	write := curBlockSize
+	for _, block := range blocks {
+		if write < len(block) {
+			data = append(data, block[:write]...)
+			return data, nil
+		}
+		data = append(data, block...)
+		write -= len(block)
+	}
+	return data, nil
+}
+
+// checkBlockSize return the size of a single block.
+// The first non-zero size is returned,
+// or 0 if all blocks are size 0.
+func checkBlockSize(blocks [][]byte) int {
+	for _, block := range blocks {
+		if len(block) != 0 {
+			return len(block)
+		}
+	}
+	return 0
+}
+
+// calculate the blockSize based on input length and total number of
+// data blocks.
+func getEncodedBlockLen(inputLen int64, dataBlocks int) (curEncBlockSize int64) {
+	curEncBlockSize = (inputLen + int64(dataBlocks) - 1) / int64(dataBlocks)
+	return curEncBlockSize
+}

erasure-waitcloser.go

@@ -0,0 +1,74 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"io"
+	"sync"
+)
+
+// waitCloser implements a Closer that blocks until released, this
+// prevents the writer closing one end of a pipe while making sure
+// that all data is written and committed to disk on the end.
+// Additionally this also implements Write().
+type waitCloser struct {
+	wg     *sync.WaitGroup // Waitgroup for atomicity.
+	writer io.WriteCloser  // Embedded writer.
+}
+
+// Write to the underlying writer.
+func (b *waitCloser) Write(data []byte) (int, error) {
+	return b.writer.Write(data)
+}
+
+// Close blocks until another goroutine calls Release(error). Returns
+// error code if either writer fails or Release is called with an error.
+func (b *waitCloser) Close() error {
+	err := b.writer.Close()
+	b.wg.Wait()
+	return err
+}
+
+// CloseWithError closes the writer; subsequent read to the read
+// half of the pipe will return the error err.
+func (b *waitCloser) CloseWithError(err error) error {
+	w, ok := b.writer.(*io.PipeWriter)
+	if ok {
+		return w.CloseWithError(err)
+	}
+	return err
+}
+
+// release the Close, causing it to unblock. Only call this
+// once. Calling it multiple times results in a panic.
+func (b *waitCloser) release() {
+	b.wg.Done()
+	return
+}
+
+// newWaitCloser creates a new write closer that must be
+// released by the read consumer.
+func newWaitCloser(writer io.WriteCloser) *waitCloser {
+	// Wait group for the go-routine.
+	wg := &sync.WaitGroup{}
+	// Add to the wait group to wait for.
+	wg.Add(1)
+	return &waitCloser{
+		wg:     wg,
+		writer: writer,
+	}
+}

erasure.go

@@ -1,7 +1,5 @@
-// +build windows
-
 /*
- * Minio Cloud Storage, (C) 2015 Minio, Inc.
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -16,19 +14,4 @@
  * limitations under the License.
  */
 
-package minhttp
-
-import (
-	"os"
-	"syscall"
-)
-
-func terminateProcess(pid, exitcode int) error {
-	h, e := syscall.OpenProcess(syscall.PROCESS_TERMINATE, false, uint32(pid))
-	if e != nil {
-		return os.NewSyscallError("OpenProcess", e)
-	}
-	defer syscall.CloseHandle(h)
-	e = syscall.TerminateProcess(h, uint32(exitcode))
-	return os.NewSyscallError("TerminateProcess", e)
-}
+package main

format-config-v1.go

@@ -0,0 +1,588 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+
+	"github.com/skyrings/skyring-common/tools/uuid"
+)
+
+// fsFormat - structure holding 'fs' format.
+type fsFormat struct {
+	Version string `json:"version"`
+}
+
+// xlFormat - structure holding 'xl' format.
+type xlFormat struct {
+	Version string `json:"version"` // Version of 'xl' format.
+	Disk    string `json:"disk"`    // Disk field carries assigned disk uuid.
+	// JBOD field carries the input disk order generated the first
+	// time when fresh disks were supplied.
+	JBOD []string `json:"jbod"`
+}
+
+// formatConfigV1 - structure holds format config version '1'.
+type formatConfigV1 struct {
+	Version string `json:"version"` // Version of the format config.
+	// Format indicates the backend format type, supports two values 'xl' and 'fs'.
+	Format string    `json:"format"`
+	FS     *fsFormat `json:"fs,omitempty"` // FS field holds fs format.
+	XL     *xlFormat `json:"xl,omitempty"` // XL field holds xl format.
+}
+
+/*
+
+All disks online
+-----------------
+- All Unformatted - format all and return success.
+- Some Unformatted - format all and return success.
+- Any JBOD inconsistent - return failure // Requires deep inspection, phase2.
+- Some are corrupt (missing format.json) - return failure  // Requires deep inspection, phase2.
+- Any unrecognized disks - return failure
+
+Some disks are offline and we have quorum.
+-----------------
+- Some unformatted - no heal, return success.
+- Any JBOD inconsistent - return failure // Requires deep inspection, phase2.
+- Some are corrupt (missing format.json) - return failure  // Requires deep inspection, phase2.
+- Any unrecognized disks - return failure
+
+No read quorum
+-----------------
+failure for all cases.
+
+// Pseudo code for managing `format.json`.
+
+// Generic checks.
+if (no quorum) return error
+if (any disk is corrupt) return error // phase2
+if (jbod inconsistent) return error // phase2
+if (disks not recognized) // Always error.
+
+// Specific checks.
+if (all disks online)
+  if (all disks return format.json)
+     if (jbod consistent)
+        if (all disks recognized)
+          return
+  else
+     if (all disks return format.json not found)
+        (initialize format)
+        return
+     else (some disks return format.json not found)
+        (heal format)
+        return
+     fi
+   fi
+else // No healing at this point forward, some disks are offline or dead.
+   if (some disks return format.json not found)
+      if (with force)
+         // Offline disks are marked as dead.
+         (heal format) // Offline disks should be marked as dead.
+         return success
+      else (without force)
+         // --force is necessary to heal few drives, because some drives
+         // are offline. Offline disks will be marked as dead.
+         return error
+      fi
+fi
+*/
+
+var errSomeDiskUnformatted = errors.New("some disks are found to be unformatted")
+var errSomeDiskOffline = errors.New("some disks are offline")
+
+// Returns error slice into understandable errors.
+func reduceFormatErrs(errs []error, diskCount int) error {
+	var errUnformattedDiskCount = 0
+	var errDiskNotFoundCount = 0
+	for _, err := range errs {
+		if err == errUnformattedDisk {
+			errUnformattedDiskCount++
+		} else if err == errDiskNotFound {
+			errDiskNotFoundCount++
+		}
+	}
+	// Returns errUnformattedDisk if all disks report unFormattedDisk.
+	if errUnformattedDiskCount == diskCount {
+		return errUnformattedDisk
+	} else if errUnformattedDiskCount < diskCount && errDiskNotFoundCount == 0 {
+		// Only some disks return unFormattedDisk and all disks are online.
+		return errSomeDiskUnformatted
+	} else if errUnformattedDiskCount < diskCount && errDiskNotFoundCount > 0 {
+		// Only some disks return unFormattedDisk and some disks are
+		// offline as well.
+		return errSomeDiskOffline
+	}
+	return nil
+}
+
+// loadAllFormats - load all format config from all input disks in parallel.
+func loadAllFormats(bootstrapDisks []StorageAPI) ([]*formatConfigV1, []error) {
+	// Initialize sync waitgroup.
+	var wg = &sync.WaitGroup{}
+
+	// Initialize list of errors.
+	var sErrs = make([]error, len(bootstrapDisks))
+
+	// Initialize format configs.
+	var formatConfigs = make([]*formatConfigV1, len(bootstrapDisks))
+
+	// Make a volume entry on all underlying storage disks.
+	for index, disk := range bootstrapDisks {
+		wg.Add(1)
+		// Make a volume inside a go-routine.
+		go func(index int, disk StorageAPI) {
+			defer wg.Done()
+			formatConfig, lErr := loadFormat(disk)
+			if lErr != nil {
+				sErrs[index] = lErr
+				return
+			}
+			formatConfigs[index] = formatConfig
+		}(index, disk)
+	}
+
+	// Wait for all make vol to finish.
+	wg.Wait()
+
+	for _, err := range sErrs {
+		if err != nil {
+			// Return all formats and errors.
+			return formatConfigs, sErrs
+		}
+	}
+	// Return all formats and nil
+	return formatConfigs, nil
+}
+
+// genericFormatCheck - validates and returns error.
+// if (no quorum) return error
+// if (any disk is corrupt) return error // phase2
+// if (jbod inconsistent) return error // phase2
+// if (disks not recognized) // Always error.
+func genericFormatCheck(formatConfigs []*formatConfigV1, sErrs []error) (err error) {
+	// Calculate the errors.
+	var (
+		errCorruptFormatCount = 0
+		errCount              = 0
+	)
+
+	// Through all errors calculate the actual errors.
+	for _, lErr := range sErrs {
+		if lErr == nil {
+			continue
+		}
+		// These errors are good conditions, means disk is online.
+		if lErr == errUnformattedDisk || lErr == errVolumeNotFound {
+			continue
+		}
+		if lErr == errCorruptedFormat {
+			errCorruptFormatCount++
+		} else {
+			errCount++
+		}
+	}
+
+	// Calculate read quorum.
+	readQuorum := len(formatConfigs)/2 + 1
+
+	// Validate the err count under tolerant limit.
+	if errCount > len(formatConfigs)-readQuorum {
+		return errXLReadQuorum
+	}
+
+	// One of the disk has corrupt format, return error.
+	if errCorruptFormatCount > 0 {
+		return errCorruptedFormat
+	}
+
+	// Validates if format and JBOD are consistent across all disks.
+	if err = checkFormatXL(formatConfigs); err != nil {
+		return err
+	}
+
+	// Success..
+	return nil
+}
+
+// checkDisksConsistency - checks if all disks are consistent with all JBOD entries on all disks.
+func checkDisksConsistency(formatConfigs []*formatConfigV1) error {
+	var disks = make([]string, len(formatConfigs))
+	var disksFound = make(map[string]bool)
+	// Collect currently available disk uuids.
+	for index, formatConfig := range formatConfigs {
+		if formatConfig == nil {
+			continue
+		}
+		disks[index] = formatConfig.XL.Disk
+	}
+	// Validate collected uuids and verify JBOD.
+	for index, uuid := range disks {
+		if uuid == "" {
+			continue
+		}
+		var formatConfig = formatConfigs[index]
+		for _, savedUUID := range formatConfig.XL.JBOD {
+			if savedUUID == uuid {
+				disksFound[uuid] = true
+			}
+		}
+	}
+	// Check if all disks are found.
+	for _, value := range disksFound {
+		if !value {
+			return errors.New("Some disks not found in JBOD.")
+		}
+	}
+	return nil
+}
+
+// checkJBODConsistency - validate xl jbod order if they are consistent.
+func checkJBODConsistency(formatConfigs []*formatConfigV1) error {
+	var firstJBOD []string
+	// Extract first valid JBOD.
+	for _, format := range formatConfigs {
+		if format == nil {
+			continue
+		}
+		firstJBOD = format.XL.JBOD
+		break
+	}
+	jbodStr := strings.Join(firstJBOD, ".")
+	for _, format := range formatConfigs {
+		if format == nil {
+			continue
+		}
+		savedJBODStr := strings.Join(format.XL.JBOD, ".")
+		if jbodStr != savedJBODStr {
+			return errors.New("Inconsistent JBOD found.")
+		}
+	}
+	return nil
+}
+
+func findIndex(disk string, jbod []string) int {
+	for index, uuid := range jbod {
+		if uuid == disk {
+			return index
+		}
+	}
+	return -1
+}
+
+// reorderDisks - reorder disks in JBOD order.
+func reorderDisks(bootstrapDisks []StorageAPI, formatConfigs []*formatConfigV1) ([]StorageAPI, error) {
+	var savedJBOD []string
+	for _, format := range formatConfigs {
+		if format == nil {
+			continue
+		}
+		savedJBOD = format.XL.JBOD
+		break
+	}
+	// Pick the first JBOD list to verify the order and construct new set of disk slice.
+	var newDisks = make([]StorageAPI, len(bootstrapDisks))
+	for fIndex, format := range formatConfigs {
+		if format == nil {
+			continue
+		}
+		jIndex := findIndex(format.XL.Disk, savedJBOD)
+		if jIndex == -1 {
+			return nil, errors.New("Unrecognized uuid " + format.XL.Disk + " found")
+		}
+		newDisks[jIndex] = bootstrapDisks[fIndex]
+	}
+	return newDisks, nil
+}
+
+// loadFormat - load format from disk.
+func loadFormat(disk StorageAPI) (format *formatConfigV1, err error) {
+	var buffer []byte
+	buffer, err = readAll(disk, minioMetaBucket, formatConfigFile)
+	if err != nil {
+		// 'file not found' and 'volume not found' as
+		// same. 'volume not found' usually means its a fresh disk.
+		if err == errFileNotFound || err == errVolumeNotFound {
+			var vols []VolInfo
+			vols, err = disk.ListVols()
+			if err != nil {
+				return nil, err
+			}
+			if len(vols) > 1 {
+				// 'format.json' not found, but we found user data.
+				return nil, errCorruptedFormat
+			}
+			// No other data found, its a fresh disk.
+			return nil, errUnformattedDisk
+		}
+		return nil, err
+	}
+	format = &formatConfigV1{}
+	err = json.Unmarshal(buffer, format)
+	if err != nil {
+		return nil, err
+	}
+	return format, nil
+}
+
+// Heals any missing format.json on the drives. Returns error only for unexpected errors
+// as regular errors can be ignored since there might be enough quorum to be operational.
+func healFormatXL(bootstrapDisks []StorageAPI) error {
+	needHeal := make([]bool, len(bootstrapDisks)) // Slice indicating which drives needs healing.
+
+	formatConfigs := make([]*formatConfigV1, len(bootstrapDisks))
+	var referenceConfig *formatConfigV1
+	successCount := 0        // Tracks if we have successfully loaded all `format.json` from all disks.
+	formatNotFoundCount := 0 // Tracks if we `format.json` is not found on all disks.
+	// Loads `format.json` from all disks.
+	for index, disk := range bootstrapDisks {
+		formatXL, err := loadFormat(disk)
+		if err != nil {
+			if err == errUnformattedDisk {
+				// format.json is missing, should be healed.
+				needHeal[index] = true
+				formatNotFoundCount++
+				continue
+			} else if err == errDiskNotFound { // Is a valid case we
+				// can proceed without healing.
+				return nil
+			}
+			// Return error for unsupported errors.
+			return err
+		} // Success.
+		formatConfigs[index] = formatXL
+		successCount++
+	}
+	// All `format.json` has been read successfully, previously completed.
+	if successCount == len(bootstrapDisks) {
+		// Return success.
+		return nil
+	}
+	// All disks are fresh, format.json will be written by initFormatXL()
+	if formatNotFoundCount == len(bootstrapDisks) {
+		return initFormatXL(bootstrapDisks)
+	}
+	// Validate format configs for consistency in JBOD and disks.
+	if err := checkFormatXL(formatConfigs); err != nil {
+		return err
+	}
+
+	if referenceConfig == nil {
+		// This config will be used to update the drives missing format.json.
+		for _, formatConfig := range formatConfigs {
+			if formatConfig == nil {
+				continue
+			}
+			referenceConfig = formatConfig
+			break
+		}
+	}
+
+	uuidUsage := make([]struct {
+		uuid  string // Disk uuid
+		inUse bool   // indicates if the uuid is used by
+		// any disk
+	}, len(bootstrapDisks))
+
+	// Returns any unused drive UUID.
+	getUnusedUUID := func() string {
+		for index := range uuidUsage {
+			if !uuidUsage[index].inUse {
+				uuidUsage[index].inUse = true
+				return uuidUsage[index].uuid
+			}
+		}
+		return ""
+	}
+
+	// From reference config update UUID's not be in use.
+	for index, diskUUID := range referenceConfig.XL.JBOD {
+		uuidUsage[index].uuid = diskUUID
+		uuidUsage[index].inUse = false
+	}
+
+	// For all config formats validate if they are in use and
+	// update the uuidUsage values.
+	for _, config := range formatConfigs {
+		if config == nil {
+			continue
+		}
+		for index := range uuidUsage {
+			if config.XL.Disk == uuidUsage[index].uuid {
+				uuidUsage[index].inUse = true
+				break
+			}
+		}
+	}
+	// This section heals the format.json and updates the fresh disks
+	// by apply a new UUID for all the fresh disks.
+	for index, heal := range needHeal {
+		if !heal {
+			continue
+		}
+		config := &formatConfigV1{}
+		*config = *referenceConfig
+		config.XL.Disk = getUnusedUUID()
+		if config.XL.Disk == "" {
+			// getUnusedUUID() should have
+			// returned an unused uuid, it
+			// is an unexpected error.
+			return errUnexpected
+		}
+
+		formatBytes, err := json.Marshal(config)
+		if err != nil {
+			return err
+		}
+		// Fresh disk without format.json
+		_, _ = bootstrapDisks[index].AppendFile(minioMetaBucket, formatConfigFile, formatBytes)
+		// Ignore any error from AppendFile() as
+		// quorum might still be there to be operational.
+	}
+	return nil
+}
+
+// loadFormatXL - loads XL `format.json` and returns back properly
+// ordered storage slice based on `format.json`.
+func loadFormatXL(bootstrapDisks []StorageAPI) (disks []StorageAPI, err error) {
+	var unformattedDisksFoundCnt = 0
+	var diskNotFoundCount = 0
+	formatConfigs := make([]*formatConfigV1, len(bootstrapDisks))
+
+	// Try to load `format.json` bootstrap disks.
+	for index, disk := range bootstrapDisks {
+		var formatXL *formatConfigV1
+		formatXL, err = loadFormat(disk)
+		if err != nil {
+			if err == errUnformattedDisk {
+				unformattedDisksFoundCnt++
+				continue
+			} else if err == errDiskNotFound {
+				diskNotFoundCount++
+				continue
+			}
+			return nil, err
+		}
+		// Save valid formats.
+		formatConfigs[index] = formatXL
+	}
+
+	// If all disks indicate that 'format.json' is not available
+	// return 'errUnformattedDisk'.
+	if unformattedDisksFoundCnt == len(bootstrapDisks) {
+		return nil, errUnformattedDisk
+	} else if diskNotFoundCount == len(bootstrapDisks) {
+		return nil, errDiskNotFound
+	} else if diskNotFoundCount > len(bootstrapDisks)-(len(bootstrapDisks)/2+1) {
+		return nil, errXLReadQuorum
+	} else if unformattedDisksFoundCnt > len(bootstrapDisks)-(len(bootstrapDisks)/2+1) {
+		return nil, errXLReadQuorum
+	}
+
+	// Validate the format configs read are correct.
+	if err = checkFormatXL(formatConfigs); err != nil {
+		return nil, err
+	}
+	// Erasure code requires disks to be presented in the same order each time.
+	return reorderDisks(bootstrapDisks, formatConfigs)
+}
+
+// checkFormatXL - verifies if format.json format is intact.
+func checkFormatXL(formatConfigs []*formatConfigV1) error {
+	for _, formatXL := range formatConfigs {
+		if formatXL == nil {
+			continue
+		}
+		// Validate format version and format type.
+		if formatXL.Version != "1" {
+			return fmt.Errorf("Unsupported version of backend format [%s] found.", formatXL.Version)
+		}
+		if formatXL.Format != "xl" {
+			return fmt.Errorf("Unsupported backend format [%s] found.", formatXL.Format)
+		}
+		if formatXL.XL.Version != "1" {
+			return fmt.Errorf("Unsupported XL backend format found [%s]", formatXL.XL.Version)
+		}
+		if len(formatConfigs) != len(formatXL.XL.JBOD) {
+			return fmt.Errorf("Number of disks %d did not match the backend format %d", len(formatConfigs), len(formatXL.XL.JBOD))
+		}
+	}
+	if err := checkJBODConsistency(formatConfigs); err != nil {
+		return err
+	}
+	return checkDisksConsistency(formatConfigs)
+}
+
+// initFormatXL - save XL format configuration on all disks.
+func initFormatXL(storageDisks []StorageAPI) (err error) {
+	var (
+		jbod             = make([]string, len(storageDisks))
+		formats          = make([]*formatConfigV1, len(storageDisks))
+		saveFormatErrCnt = 0
+	)
+	for index, disk := range storageDisks {
+		if err = disk.MakeVol(minioMetaBucket); err != nil {
+			if err != errVolumeExists {
+				saveFormatErrCnt++
+				// Check for write quorum.
+				if saveFormatErrCnt <= len(storageDisks)-(len(storageDisks)/2+3) {
+					continue
+				}
+				return errXLWriteQuorum
+			}
+		}
+		var u *uuid.UUID
+		u, err = uuid.New()
+		if err != nil {
+			saveFormatErrCnt++
+			// Check for write quorum.
+			if saveFormatErrCnt <= len(storageDisks)-(len(storageDisks)/2+3) {
+				continue
+			}
+			return err
+		}
+		formats[index] = &formatConfigV1{
+			Version: "1",
+			Format:  "xl",
+			XL: &xlFormat{
+				Version: "1",
+				Disk:    u.String(),
+			},
+		}
+		jbod[index] = formats[index].XL.Disk
+	}
+	for index, disk := range storageDisks {
+		formats[index].XL.JBOD = jbod
+		formatBytes, err := json.Marshal(formats[index])
+		if err != nil {
+			return err
+		}
+		n, err := disk.AppendFile(minioMetaBucket, formatConfigFile, formatBytes)
+		if err != nil {
+			return err
+		}
+		if n != int64(len(formatBytes)) {
+			return errUnexpected
+		}
+	}
+	return nil
+}

fs-bucket-listobjects.go

@@ -1,182 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015-2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-
-	"github.com/minio/minio/pkg/probe"
-)
-
-const (
-	// listObjectsLimit - maximum list objects limit.
-	listObjectsLimit = 1000
-)
-
-// isDirExist - returns whether given directory is exist or not.
-func isDirExist(dirname string) (bool, error) {
-	fi, e := os.Lstat(dirname)
-	if e != nil {
-		if os.IsNotExist(e) {
-			return false, nil
-		}
-		return false, e
-	}
-	return fi.IsDir(), nil
-}
-
-func (fs *Filesystem) saveTreeWalk(params listObjectParams, walker *treeWalker) {
-	fs.listObjectMapMutex.Lock()
-	defer fs.listObjectMapMutex.Unlock()
-
-	walkers, _ := fs.listObjectMap[params]
-	walkers = append(walkers, walker)
-
-	fs.listObjectMap[params] = walkers
-}
-
-func (fs *Filesystem) lookupTreeWalk(params listObjectParams) *treeWalker {
-	fs.listObjectMapMutex.Lock()
-	defer fs.listObjectMapMutex.Unlock()
-
-	if walkChs, ok := fs.listObjectMap[params]; ok {
-		for i, walkCh := range walkChs {
-			if !walkCh.timedOut {
-				newWalkChs := walkChs[i+1:]
-				if len(newWalkChs) > 0 {
-					fs.listObjectMap[params] = newWalkChs
-				} else {
-					delete(fs.listObjectMap, params)
-				}
-				return walkCh
-			}
-		}
-		// As all channels are timed out, delete the map entry
-		delete(fs.listObjectMap, params)
-	}
-	return nil
-}
-
-// ListObjects - lists all objects for a given prefix, returns up to
-// maxKeys number of objects per call.
-func (fs Filesystem) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) {
-	result := ListObjectsInfo{}
-
-	// Input validation.
-	bucket, e := fs.checkBucketArg(bucket)
-	if e != nil {
-		return result, probe.NewError(e)
-	}
-	bucketDir := filepath.Join(fs.diskPath, bucket)
-
-	if !IsValidObjectPrefix(prefix) {
-		return result, probe.NewError(ObjectNameInvalid{Bucket: bucket, Object: prefix})
-	}
-
-	// Verify if delimiter is anything other than '/', which we do not support.
-	if delimiter != "" && delimiter != "/" {
-		return result, probe.NewError(fmt.Errorf("delimiter '%s' is not supported. Only '/' is supported", delimiter))
-	}
-
-	// Verify if marker has prefix.
-	if marker != "" {
-		if !strings.HasPrefix(marker, prefix) {
-			return result, probe.NewError(fmt.Errorf("Invalid combination of marker '%s' and prefix '%s'", marker, prefix))
-		}
-	}
-
-	// Return empty response for a valid request when maxKeys is 0.
-	if maxKeys == 0 {
-		return result, nil
-	}
-
-	// Over flowing maxkeys - reset to listObjectsLimit.
-	if maxKeys < 0 || maxKeys > listObjectsLimit {
-		maxKeys = listObjectsLimit
-	}
-
-	// Verify if prefix exists.
-	prefixDir := filepath.Dir(filepath.FromSlash(prefix))
-	rootDir := filepath.Join(bucketDir, prefixDir)
-	if status, e := isDirExist(rootDir); !status {
-		if e == nil {
-			// Prefix does not exist, not an error just respond empty
-			// list response.
-			return result, nil
-		}
-		// Rest errors should be treated as failure.
-		return result, probe.NewError(e)
-	}
-
-	recursive := true
-	if delimiter == "/" {
-		recursive = false
-	}
-
-	// Maximum 1000 objects returned in a single to listObjects.
-	// Further calls will set right marker value to continue reading the rest of the objectList.
-	// popTreeWalker returns nil if the call to ListObject is done for the first time.
-	// On further calls to ListObjects to retrive more objects within the timeout period,
-	// popTreeWalker returns the channel from which rest of the objects can be retrieved.
-	walker := fs.lookupTreeWalk(listObjectParams{bucket, delimiter, marker, prefix})
-	if walker == nil {
-		walker = startTreeWalk(fs.diskPath, bucket, filepath.FromSlash(prefix), filepath.FromSlash(marker), recursive)
-	}
-
-	nextMarker := ""
-	for i := 0; i < maxKeys; {
-		walkResult, ok := <-walker.ch
-		if !ok {
-			// Closed channel.
-			return result, nil
-		}
-		// For any walk error return right away.
-		if walkResult.err != nil {
-			return ListObjectsInfo{}, probe.NewError(walkResult.err)
-		}
-		objInfo := walkResult.objectInfo
-		objInfo.Name = filepath.ToSlash(objInfo.Name)
-
-		// Skip temporary files.
-		if strings.Contains(objInfo.Name, "$multiparts") || strings.Contains(objInfo.Name, "$tmpobject") {
-			continue
-		}
-
-		// For objects being directory and delimited we set Prefixes.
-		if objInfo.IsDir {
-			result.Prefixes = append(result.Prefixes, objInfo.Name)
-		} else {
-			result.Objects = append(result.Objects, objInfo)
-		}
-
-		// We have listed everything return.
-		if walkResult.end {
-			return result, nil
-		}
-		nextMarker = objInfo.Name
-		i++
-	}
-	// We haven't exhaused the list yet, set IsTruncated to 'true' so
-	// that the client can send another request.
-	result.IsTruncated = true
-	result.NextMarker = nextMarker
-	fs.saveTreeWalk(listObjectParams{bucket, delimiter, nextMarker, prefix}, walker)
-	return result, nil
-}

fs-bucket.go

@@ -1,163 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"io/ioutil"
-	"os"
-	"path/filepath"
-	"strings"
-
-	"github.com/minio/minio/pkg/probe"
-)
-
-/// Bucket Operations
-
-// DeleteBucket - delete a bucket.
-func (fs Filesystem) DeleteBucket(bucket string) *probe.Error {
-	// Verify bucket is valid.
-	if !IsValidBucketName(bucket) {
-		return probe.NewError(BucketNameInvalid{Bucket: bucket})
-	}
-	bucket = getActualBucketname(fs.diskPath, bucket)
-	bucketDir := filepath.Join(fs.diskPath, bucket)
-	if e := os.Remove(bucketDir); e != nil {
-		// Error if there was no bucket in the first place.
-		if os.IsNotExist(e) {
-			return probe.NewError(BucketNotFound{Bucket: bucket})
-		}
-		// On windows the string is slightly different, handle it here.
-		if strings.Contains(e.Error(), "directory is not empty") {
-			return probe.NewError(BucketNotEmpty{Bucket: bucket})
-		}
-		// Hopefully for all other operating systems, this is
-		// assumed to be consistent.
-		if strings.Contains(e.Error(), "directory not empty") {
-			return probe.NewError(BucketNotEmpty{Bucket: bucket})
-		}
-		return probe.NewError(e)
-	}
-	return nil
-}
-
-// ListBuckets - Get service.
-func (fs Filesystem) ListBuckets() ([]BucketInfo, *probe.Error) {
-	files, e := ioutil.ReadDir(fs.diskPath)
-	if e != nil {
-		return []BucketInfo{}, probe.NewError(e)
-	}
-	var buckets []BucketInfo
-	for _, file := range files {
-		if !file.IsDir() {
-			// If not directory, ignore all file types.
-			continue
-		}
-		// If directories are found with odd names, skip them.
-		dirName := strings.ToLower(file.Name())
-		if !IsValidBucketName(dirName) {
-			continue
-		}
-		bucket := BucketInfo{
-			Name:    dirName,
-			Created: file.ModTime(),
-		}
-		buckets = append(buckets, bucket)
-	}
-	// Remove duplicated entries.
-	buckets = removeDuplicateBuckets(buckets)
-	return buckets, nil
-}
-
-// removeDuplicateBuckets - remove duplicate buckets.
-func removeDuplicateBuckets(buckets []BucketInfo) []BucketInfo {
-	length := len(buckets) - 1
-	for i := 0; i < length; i++ {
-		for j := i + 1; j <= length; j++ {
-			if buckets[i].Name == buckets[j].Name {
-				if buckets[i].Created.Sub(buckets[j].Created) > 0 {
-					buckets[i] = buckets[length]
-				} else {
-					buckets[j] = buckets[length]
-				}
-				buckets = buckets[0:length]
-				length--
-				j--
-			}
-		}
-	}
-	return buckets
-}
-
-// MakeBucket - PUT Bucket
-func (fs Filesystem) MakeBucket(bucket string) *probe.Error {
-	if _, e := fs.checkBucketArg(bucket); e == nil {
-		return probe.NewError(BucketExists{Bucket: bucket})
-	} else if _, ok := e.(BucketNameInvalid); ok {
-		return probe.NewError(BucketNameInvalid{Bucket: bucket})
-	}
-	bucketDir := filepath.Join(fs.diskPath, bucket)
-
-	// Make bucket.
-	if e := os.Mkdir(bucketDir, 0700); e != nil {
-		return probe.NewError(e)
-	}
-	return nil
-}
-
-// getActualBucketname - will convert incoming bucket names to
-// corresponding actual bucketnames on the backend in a platform
-// compatible way for all operating systems.
-func getActualBucketname(fsPath, bucket string) string {
-	fd, e := os.Open(fsPath)
-	if e != nil {
-		return bucket
-	}
-	buckets, e := fd.Readdirnames(-1)
-	if e != nil {
-		return bucket
-	}
-	for _, b := range buckets {
-		// Verify if lowercase version of the bucket is equal
-		// to the incoming bucket, then use the proper name.
-		if strings.ToLower(b) == bucket {
-			return b
-		}
-	}
-	return bucket
-}
-
-// GetBucketInfo - get bucket metadata.
-func (fs Filesystem) GetBucketInfo(bucket string) (BucketInfo, *probe.Error) {
-	if !IsValidBucketName(bucket) {
-		return BucketInfo{}, probe.NewError(BucketNameInvalid{Bucket: bucket})
-	}
-	bucket = getActualBucketname(fs.diskPath, bucket)
-	// Get bucket path.
-	bucketDir := filepath.Join(fs.diskPath, bucket)
-	fi, e := os.Stat(bucketDir)
-	if e != nil {
-		// Check if bucket exists.
-		if os.IsNotExist(e) {
-			return BucketInfo{}, probe.NewError(BucketNotFound{Bucket: bucket})
-		}
-		return BucketInfo{}, probe.NewError(e)
-	}
-	bucketMetadata := BucketInfo{}
-	bucketMetadata.Name = fi.Name()
-	bucketMetadata.Created = fi.ModTime()
-	return bucketMetadata, nil
-}

fs-bucket_test.go

@@ -1,256 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"io/ioutil"
-	"os"
-	"strconv"
-	"strings"
-	"testing"
-)
-
-// The test not just includes asserting the correctness of the output,
-// But also includes test cases for which the function should fail.
-// For those cases for which it fails, its also asserted whether the function fails as expected.
-func TestGetBucketInfo(t *testing.T) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-metadata-test")
-	if e != nil {
-		t.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Creating few buckets.
-	for i := 0; i < 4; i++ {
-		err = fs.MakeBucket("meta-test-bucket." + strconv.Itoa(i))
-		if err != nil {
-			t.Fatal(err)
-		}
-	}
-	testCases := []struct {
-		bucketName string
-		metaData   BucketInfo
-		e          error
-		shouldPass bool
-	}{
-		// Test cases with invalid bucket names.
-		{".test", BucketInfo{}, BucketNameInvalid{Bucket: ".test"}, false},
-		{"Test", BucketInfo{}, BucketNameInvalid{Bucket: "Test"}, false},
-		{"---", BucketInfo{}, BucketNameInvalid{Bucket: "---"}, false},
-		{"ad", BucketInfo{}, BucketNameInvalid{Bucket: "ad"}, false},
-		// Test cases with non-existent buckets.
-		{"volatile-bucket-1", BucketInfo{}, BucketNotFound{Bucket: "volatile-bucket-1"}, false},
-		{"volatile-bucket-2", BucketInfo{}, BucketNotFound{Bucket: "volatile-bucket-2"}, false},
-		// Test cases with existing buckets.
-		{"meta-test-bucket.0", BucketInfo{Name: "meta-test-bucket.0"}, nil, true},
-		{"meta-test-bucket.1", BucketInfo{Name: "meta-test-bucket.1"}, nil, true},
-		{"meta-test-bucket.2", BucketInfo{Name: "meta-test-bucket.2"}, nil, true},
-		{"meta-test-bucket.3", BucketInfo{Name: "meta-test-bucket.3"}, nil, true},
-	}
-	for i, testCase := range testCases {
-		// The err returned is of type *probe.Error.
-		bucketInfo, err := fs.GetBucketInfo(testCase.bucketName)
-
-		if err != nil && testCase.shouldPass {
-			t.Errorf("Test %d: Expected to pass, but failed with: <ERROR> %s", i+1, err.Cause.Error())
-		}
-		if err == nil && !testCase.shouldPass {
-			t.Errorf("Test %d: Expected to fail with <ERROR> \"%s\", but passed instead", i+1, testCase.e.Error())
-
-		}
-		// Failed as expected, but does it fail for the expected reason.
-		if err != nil && !testCase.shouldPass {
-			if testCase.e.Error() != err.Cause.Error() {
-				t.Errorf("Test %d: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead", i+1, testCase.e.Error(), err.Cause.Error())
-			}
-		}
-		// Since there are cases for which GetBucketInfo fails, this is necessary.
-		// Test passes as expected, but the output values are verified for correctness here.
-		if err == nil && testCase.shouldPass {
-			if testCase.bucketName != bucketInfo.Name {
-				t.Errorf("Test %d: Expected the bucket name to be \"%s\", but found \"%s\" instead", i+1, testCase.bucketName, bucketInfo.Name)
-			}
-		}
-	}
-}
-
-func TestListBuckets(t *testing.T) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-benchmark")
-	if e != nil {
-		t.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Create a few buckets.
-	for i := 0; i < 10; i++ {
-		err = fs.MakeBucket("testbucket." + strconv.Itoa(i))
-		if err != nil {
-			t.Fatal(err)
-		}
-	}
-
-	// List, and ensure that they are all there.
-	metadatas, err := fs.ListBuckets()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if len(metadatas) != 10 {
-		t.Errorf("incorrect length of metadatas (%d)\n", len(metadatas))
-	}
-
-	// Iterate over the buckets, ensuring that the name is correct.
-	for i := 0; i < len(metadatas); i++ {
-		if !strings.Contains(metadatas[i].Name, "testbucket") {
-			t.Fail()
-		}
-	}
-}
-
-func TestDeleteBucket(t *testing.T) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-benchmark")
-	if e != nil {
-		t.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Deleting a bucket that doesn't exist should error.
-	err = fs.DeleteBucket("bucket")
-	if !strings.Contains(err.Cause.Error(), "Bucket not found:") {
-		t.Fail()
-	}
-}
-
-func BenchmarkListBuckets(b *testing.B) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-benchmark")
-	if e != nil {
-		b.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		b.Fatal(err)
-	}
-
-	// Create a few buckets.
-	for i := 0; i < 20; i++ {
-		err = fs.MakeBucket("bucket." + strconv.Itoa(i))
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-
-	b.ResetTimer()
-
-	// List the buckets over and over and over.
-	for i := 0; i < b.N; i++ {
-		_, err = fs.ListBuckets()
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}
-
-func BenchmarkDeleteBucket(b *testing.B) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-benchmark")
-	if e != nil {
-		b.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		b.Fatal(err)
-	}
-
-	b.ResetTimer()
-
-	for i := 0; i < b.N; i++ {
-		// Creating buckets takes time, so stop and start the timer.
-		b.StopTimer()
-
-		// Create and delete the bucket over and over.
-		err = fs.MakeBucket("bucket")
-		if err != nil {
-			b.Fatal(err)
-		}
-
-		b.StartTimer()
-
-		err = fs.DeleteBucket("bucket")
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}
-
-func BenchmarkGetBucketInfo(b *testing.B) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-benchmark")
-	if e != nil {
-		b.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		b.Fatal(err)
-	}
-
-	// Put up a bucket with some metadata.
-	err = fs.MakeBucket("bucket")
-	if err != nil {
-		b.Fatal(err)
-	}
-
-	b.ResetTimer()
-
-	for i := 0; i < b.N; i++ {
-		// Retrieve the metadata!
-		_, err := fs.GetBucketInfo("bucket")
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-}

fs-dir-common.go

@@ -1,217 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"os"
-	"path/filepath"
-	"sort"
-	"strings"
-	"time"
-)
-
-// fsDirent carries directory entries.
-type fsDirent struct {
-	name    string
-	modTime time.Time // On Solaris and older unix distros this is empty.
-	size    int64     // On Solaris and older unix distros this is empty.
-	mode    os.FileMode
-}
-
-// IsDir - returns true if fsDirent is a directory
-func (ent fsDirent) IsDir() bool {
-	return ent.mode.IsDir()
-}
-
-// IsSymlink - returns true if fsDirent is a symbolic link
-func (ent fsDirent) IsSymlink() bool {
-	return ent.mode&os.ModeSymlink == os.ModeSymlink
-}
-
-// IsRegular - returns true if fsDirent is a regular file
-func (ent fsDirent) IsRegular() bool {
-	return ent.mode.IsRegular()
-}
-
-// byDirentName is a collection satisfying sort.Interface.
-type byDirentName []fsDirent
-
-func (d byDirentName) Len() int           { return len(d) }
-func (d byDirentName) Swap(i, j int)      { d[i], d[j] = d[j], d[i] }
-func (d byDirentName) Less(i, j int) bool { return d[i].name < d[j].name }
-
-// Using sort.Search() internally to jump to the file entry containing the prefix.
-func searchDirents(dirents []fsDirent, x string) int {
-	processFunc := func(i int) bool {
-		return dirents[i].name >= x
-	}
-	return sort.Search(len(dirents), processFunc)
-}
-
-// Tree walk result carries results of tree walking.
-type treeWalkResult struct {
-	objectInfo ObjectInfo
-	err        error
-	end        bool
-}
-
-// Tree walk notify carries a channel which notifies tree walk
-// results, additionally it also carries information if treeWalk
-// should be timedOut.
-type treeWalker struct {
-	ch       <-chan treeWalkResult
-	timedOut bool
-}
-
-// treeWalk walks FS directory tree recursively pushing ObjectInfo into the channel as and when it encounters files.
-func treeWalk(bucketDir, prefixDir, entryPrefixMatch, marker string, recursive bool, send func(treeWalkResult) bool, count *int) bool {
-	// Example:
-	// if prefixDir="one/two/three/" and marker="four/five.txt" treeWalk is recursively
-	// called with prefixDir="one/two/three/four/" and marker="five.txt"
-
-	// Convert dirent to ObjectInfo
-	direntToObjectInfo := func(dirent fsDirent) (ObjectInfo, error) {
-		objectInfo := ObjectInfo{}
-		// Convert to full object name.
-		objectInfo.Name = filepath.Join(prefixDir, dirent.name)
-		if dirent.modTime.IsZero() && dirent.size == 0 {
-			// ModifiedTime and Size are zero, Stat() and figure out
-			// the actual values that need to be set.
-			fi, err := os.Stat(filepath.Join(bucketDir, prefixDir, dirent.name))
-			if err != nil {
-				return ObjectInfo{}, err
-			}
-			// Fill size and modtime.
-			objectInfo.ModifiedTime = fi.ModTime()
-			objectInfo.Size = fi.Size()
-			objectInfo.IsDir = fi.IsDir()
-		} else {
-			// If ModifiedTime or Size are set then use them
-			// without attempting another Stat operation.
-			objectInfo.ModifiedTime = dirent.modTime
-			objectInfo.Size = dirent.size
-			objectInfo.IsDir = dirent.IsDir()
-		}
-		if objectInfo.IsDir {
-			// Add os.PathSeparator suffix again for directories as
-			// filepath.Join would have removed it.
-			objectInfo.Size = 0
-			objectInfo.Name += string(os.PathSeparator)
-		}
-		return objectInfo, nil
-	}
-
-	var markerBase, markerDir string
-	if marker != "" {
-		// Ex: if marker="four/five.txt", markerDir="four/" markerBase="five.txt"
-		markerSplit := strings.SplitN(marker, string(os.PathSeparator), 2)
-		markerDir = markerSplit[0]
-		if len(markerSplit) == 2 {
-			markerDir += string(os.PathSeparator)
-			markerBase = markerSplit[1]
-		}
-	}
-
-	// readDirAll returns entries that begins with entryPrefixMatch
-	dirents, err := readDirAll(filepath.Join(bucketDir, prefixDir), entryPrefixMatch)
-	if err != nil {
-		send(treeWalkResult{err: err})
-		return false
-	}
-	// example:
-	// If markerDir="four/" searchDirents() returns the index of "four/" in the sorted
-	// dirents list. We skip all the dirent entries till "four/"
-	dirents = dirents[searchDirents(dirents, markerDir):]
-	*count += len(dirents)
-	for i, dirent := range dirents {
-		if i == 0 && markerDir == dirent.name && !dirent.IsDir() {
-			// If the first entry is not a directory
-			// we need to skip this entry.
-			*count--
-			continue
-		}
-		if dirent.IsDir() && recursive {
-			// If the entry is a directory, we will need recurse into it.
-			markerArg := ""
-			if dirent.name == markerDir {
-				// We need to pass "five.txt" as marker only if we are
-				// recursing into "four/"
-				markerArg = markerBase
-			}
-			*count--
-			if !treeWalk(bucketDir, filepath.Join(prefixDir, dirent.name), "", markerArg, recursive, send, count) {
-				return false
-			}
-			continue
-		}
-		objectInfo, err := direntToObjectInfo(dirent)
-		if err != nil {
-			send(treeWalkResult{err: err})
-			return false
-		}
-		*count--
-		if !send(treeWalkResult{objectInfo: objectInfo}) {
-			return false
-		}
-	}
-	return true
-}
-
-// Initiate a new treeWalk in a goroutine.
-func startTreeWalk(fsPath, bucket, prefix, marker string, recursive bool) *treeWalker {
-	// Example 1
-	// If prefix is "one/two/three/" and marker is "one/two/three/four/five.txt"
-	// treeWalk is called with prefixDir="one/two/three/" and marker="four/five.txt"
-	// and entryPrefixMatch=""
-
-	// Example 2
-	// if prefix is "one/two/th" and marker is "one/two/three/four/five.txt"
-	// treeWalk is called with prefixDir="one/two/" and marker="three/four/five.txt"
-	// and entryPrefixMatch="th"
-	ch := make(chan treeWalkResult, listObjectsLimit)
-	walkNotify := treeWalker{ch: ch}
-	entryPrefixMatch := prefix
-	prefixDir := ""
-	lastIndex := strings.LastIndex(prefix, string(os.PathSeparator))
-	if lastIndex != -1 {
-		entryPrefixMatch = prefix[lastIndex+1:]
-		prefixDir = prefix[:lastIndex+1]
-	}
-	count := 0
-	marker = strings.TrimPrefix(marker, prefixDir)
-	go func() {
-		defer close(ch)
-		send := func(walkResult treeWalkResult) bool {
-			// Add the bucket.
-			walkResult.objectInfo.Bucket = bucket
-			if count == 0 {
-				walkResult.end = true
-			}
-			timer := time.After(time.Second * 60)
-			select {
-			case ch <- walkResult:
-				return true
-			case <-timer:
-				walkNotify.timedOut = true
-				return false
-			}
-		}
-		bucketDir := filepath.Join(fsPath, bucket)
-		treeWalk(bucketDir, prefixDir, entryPrefixMatch, marker, recursive, send, &count)
-	}()
-	return &walkNotify
-}

fs-dir-others.go

@@ -1,108 +0,0 @@
-// +build !linux,!darwin,!openbsd,!freebsd,!netbsd,!dragonfly
-
-/*
- * Minio Cloud Storage, (C) 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"io"
-	"os"
-	"path/filepath"
-	"sort"
-	"strings"
-)
-
-// Read all directory entries, returns a list of lexically sorted entries.
-func readDirAll(readDirPath, entryPrefixMatch string) ([]fsDirent, error) {
-	f, err := os.Open(readDirPath)
-	if err != nil {
-		return nil, err
-	}
-	defer f.Close()
-	var dirents []fsDirent
-	for {
-		fis, err := f.Readdir(1000)
-		if err != nil {
-			if err == io.EOF {
-				break
-			}
-			return nil, err
-		}
-		for _, fi := range fis {
-			dirent := fsDirent{
-				name:    fi.Name(),
-				modTime: fi.ModTime(),
-				size:    fi.Size(),
-				mode:    fi.Mode(),
-			}
-			if dirent.IsDir() {
-				dirent.name += string(os.PathSeparator)
-				dirent.size = 0
-			}
-			if strings.HasPrefix(fi.Name(), entryPrefixMatch) {
-				dirents = append(dirents, dirent)
-			}
-		}
-	}
-	// Sort dirents.
-	sort.Sort(byDirentName(dirents))
-	return dirents, nil
-}
-
-// scans the directory dirPath, calling filter() on each directory
-// entry.  Entries for which filter() returns true are stored, lexically
-// sorted using sort.Sort(). If filter is NULL, all entries are selected.
-// If namesOnly is true, dirPath is not appended into entry name.
-func scandir(dirPath string, filter func(fsDirent) bool, namesOnly bool) ([]fsDirent, error) {
-	d, err := os.Open(dirPath)
-	if err != nil {
-		return nil, err
-	}
-	defer d.Close()
-
-	var dirents []fsDirent
-	for {
-		fis, err := d.Readdir(1000)
-		if err != nil {
-			if err == io.EOF {
-				break
-			}
-			return nil, err
-		}
-		for _, fi := range fis {
-			dirent := fsDirent{
-				name:    fi.Name(),
-				modTime: fi.ModTime(),
-				size:    fi.Size(),
-				mode:    fi.Mode(),
-			}
-			if !namesOnly {
-				dirent.name = filepath.Join(dirPath, dirent.name)
-			}
-			if dirent.IsDir() {
-				dirent.name += string(os.PathSeparator)
-			}
-			if filter == nil || filter(dirent) {
-				dirents = append(dirents, dirent)
-			}
-		}
-	}
-
-	sort.Sort(byDirentName(dirents))
-
-	return dirents, nil
-}

fs-errors.go

@@ -1,295 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import "fmt"
-
-// InvalidArgument invalid argument
-type InvalidArgument struct{}
-
-func (e InvalidArgument) Error() string {
-	return "Invalid argument"
-}
-
-// UnsupportedFilesystem unsupported filesystem type
-type UnsupportedFilesystem struct {
-	Type string
-}
-
-func (e UnsupportedFilesystem) Error() string {
-	return "Unsupported filesystem: " + e.Type
-}
-
-// RootPathFull root path out of space
-type RootPathFull struct {
-	Path string
-}
-
-func (e RootPathFull) Error() string {
-	return "Root path " + e.Path + " reached its minimum free disk threshold."
-}
-
-// BucketNotFound bucket does not exist
-type BucketNotFound struct {
-	Bucket string
-}
-
-func (e BucketNotFound) Error() string {
-	return "Bucket not found: " + e.Bucket
-}
-
-// BucketNotEmpty bucket is not empty
-type BucketNotEmpty struct {
-	Bucket string
-}
-
-func (e BucketNotEmpty) Error() string {
-	return "Bucket not empty: " + e.Bucket
-}
-
-// ObjectNotFound object does not exist
-type ObjectNotFound struct {
-	Bucket string
-	Object string
-}
-
-func (e ObjectNotFound) Error() string {
-	return "Object not found: " + e.Bucket + "#" + e.Object
-}
-
-// ObjectExistsAsPrefix object already exists with a requested prefix.
-type ObjectExistsAsPrefix struct {
-	Bucket string
-	Prefix string
-}
-
-func (e ObjectExistsAsPrefix) Error() string {
-	return "Object exists on : " + e.Bucket + " as prefix " + e.Prefix
-}
-
-// ObjectCorrupted object found to be corrupted
-type ObjectCorrupted struct {
-	Object string
-}
-
-func (e ObjectCorrupted) Error() string {
-	return "Object found corrupted: " + e.Object
-}
-
-// BucketExists bucket exists
-type BucketExists struct {
-	Bucket string
-}
-
-func (e BucketExists) Error() string {
-	return "Bucket exists: " + e.Bucket
-}
-
-// CorruptedBackend backend found to be corrupted
-type CorruptedBackend struct {
-	Backend string
-}
-
-func (e CorruptedBackend) Error() string {
-	return "Corrupted backend: " + e.Backend
-}
-
-// NotImplemented function not implemented
-type NotImplemented struct {
-	Function string
-}
-
-func (e NotImplemented) Error() string {
-	return "Not implemented: " + e.Function
-}
-
-// InvalidDisksArgument invalid number of disks per node
-type InvalidDisksArgument struct{}
-
-func (e InvalidDisksArgument) Error() string {
-	return "Invalid number of disks per node"
-}
-
-// BadDigest - Content-MD5 you specified did not match what we received.
-type BadDigest struct {
-	ExpectedMD5   string
-	CalculatedMD5 string
-}
-
-func (e BadDigest) Error() string {
-	return "Bad digest expected " + e.ExpectedMD5 + " is not valid with what we calculated " + e.CalculatedMD5
-}
-
-// InternalError - generic internal error
-type InternalError struct{}
-
-// BackendError - generic disk backend error
-type BackendError struct {
-	Path string
-}
-
-// BackendCorrupted - path has corrupted data
-type BackendCorrupted BackendError
-
-// APINotImplemented - generic API not implemented error
-type APINotImplemented struct {
-	API string
-}
-
-// GenericBucketError - generic bucket error
-type GenericBucketError struct {
-	Bucket string
-}
-
-// BucketPolicyNotFound - no bucket policy found.
-type BucketPolicyNotFound GenericBucketError
-
-func (e BucketPolicyNotFound) Error() string {
-	return "No bucket policy found for bucket: " + e.Bucket
-}
-
-// GenericObjectError - generic object error
-type GenericObjectError struct {
-	Bucket string
-	Object string
-}
-
-// ImplementationError - generic implementation error
-type ImplementationError struct {
-	Bucket string
-	Object string
-	Err    error
-}
-
-/// Bucket related errors
-
-// BucketNameInvalid - bucketname provided is invalid
-type BucketNameInvalid GenericBucketError
-
-/// Object related errors
-
-// ObjectNameInvalid - object name provided is invalid
-type ObjectNameInvalid GenericObjectError
-
-// Return string an error formatted as the given text
-func (e ImplementationError) Error() string {
-	error := ""
-	if e.Bucket != "" {
-		error = error + "Bucket: " + e.Bucket + " "
-	}
-	if e.Object != "" {
-		error = error + "Object: " + e.Object + " "
-	}
-	error = error + "Error: " + e.Err.Error()
-	return error
-}
-
-// EmbedError - wrapper function for error object
-func EmbedError(bucket, object string, err error) ImplementationError {
-	return ImplementationError{
-		Bucket: bucket,
-		Object: object,
-		Err:    err,
-	}
-}
-
-// Return string an error formatted as the given text
-func (e InternalError) Error() string {
-	return "Internal error occured"
-}
-
-// Return string an error formatted as the given text
-func (e APINotImplemented) Error() string {
-	return "Api not implemented: " + e.API
-}
-
-// Return string an error formatted as the given text
-func (e BucketNameInvalid) Error() string {
-	return "Bucket name invalid: " + e.Bucket
-}
-
-// Return string an error formatted as the given text
-func (e ObjectNameInvalid) Error() string {
-	return "Object name invalid: " + e.Bucket + "#" + e.Object
-}
-
-// IncompleteBody You did not provide the number of bytes specified by the Content-Length HTTP header
-type IncompleteBody GenericObjectError
-
-// Return string an error formatted as the given text
-func (e IncompleteBody) Error() string {
-	return e.Bucket + "#" + e.Object + "has incomplete body"
-}
-
-// Return string an error formatted as the given text
-func (e BackendCorrupted) Error() string {
-	return "Backend corrupted: " + e.Path
-}
-
-// OperationNotPermitted - operation not permitted
-type OperationNotPermitted struct {
-	Op     string
-	Reason string
-}
-
-func (e OperationNotPermitted) Error() string {
-	return "Operation " + e.Op + " not permitted for reason: " + e.Reason
-}
-
-// InvalidRange - invalid range
-type InvalidRange struct {
-	Start  int64
-	Length int64
-}
-
-func (e InvalidRange) Error() string {
-	return fmt.Sprintf("Invalid range start:%d length:%d", e.Start, e.Length)
-}
-
-/// Multipart related errors
-
-// InvalidUploadID invalid upload id
-type InvalidUploadID struct {
-	UploadID string
-}
-
-func (e InvalidUploadID) Error() string {
-	return "Invalid upload id " + e.UploadID
-}
-
-// InvalidPart One or more of the specified parts could not be found
-type InvalidPart struct{}
-
-func (e InvalidPart) Error() string {
-	return "One or more of the specified parts could not be found"
-}
-
-// InvalidPartOrder parts are not ordered as Requested
-type InvalidPartOrder struct {
-	UploadID string
-}
-
-func (e InvalidPartOrder) Error() string {
-	return "Invalid part order sent for " + e.UploadID
-}
-
-// MalformedXML invalid xml format
-type MalformedXML struct{}
-
-func (e MalformedXML) Error() string {
-	return "Malformed XML"
-}

fs-multipart-dir.go

@@ -1,280 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"errors"
-	"os"
-	"path/filepath"
-	"strings"
-	"time"
-)
-
-func scanMultipartDir(bucketDir, prefixPath, markerPath, uploadIDMarker string, recursive bool) <-chan multipartObjectInfo {
-	objectInfoCh := make(chan multipartObjectInfo, listObjectsLimit)
-
-	// TODO: check if bucketDir is absolute path
-	scanDir := bucketDir
-	dirDepth := bucketDir
-
-	if prefixPath != "" {
-		if !filepath.IsAbs(prefixPath) {
-			tmpPrefixPath := filepath.Join(bucketDir, prefixPath)
-			if strings.HasSuffix(prefixPath, string(os.PathSeparator)) {
-				tmpPrefixPath += string(os.PathSeparator)
-			}
-			prefixPath = tmpPrefixPath
-		}
-
-		// TODO: check if prefixPath starts with bucketDir
-
-		// Case #1: if prefixPath is /mnt/mys3/mybucket/2012/photos/paris, then
-		//          dirDepth is /mnt/mys3/mybucket/2012/photos
-		// Case #2: if prefixPath is /mnt/mys3/mybucket/2012/photos/, then
-		//          dirDepth is /mnt/mys3/mybucket/2012/photos
-		dirDepth = filepath.Dir(prefixPath)
-		scanDir = dirDepth
-	} else {
-		prefixPath = bucketDir
-	}
-
-	if markerPath != "" {
-		if !filepath.IsAbs(markerPath) {
-			tmpMarkerPath := filepath.Join(bucketDir, markerPath)
-			if strings.HasSuffix(markerPath, string(os.PathSeparator)) {
-				tmpMarkerPath += string(os.PathSeparator)
-			}
-
-			markerPath = tmpMarkerPath
-		}
-
-		// TODO: check markerPath must be a file
-		if uploadIDMarker != "" {
-			markerPath = filepath.Join(markerPath, uploadIDMarker+multipartUploadIDSuffix)
-		}
-
-		// TODO: check if markerPath starts with bucketDir
-		// TODO: check if markerPath starts with prefixPath
-
-		// Case #1: if markerPath is /mnt/mys3/mybucket/2012/photos/gophercon.png, then
-		//          scanDir is /mnt/mys3/mybucket/2012/photos
-		// Case #2: if markerPath is /mnt/mys3/mybucket/2012/photos/gophercon.png/1fbd117a-268a-4ed0-85c9-8cc3888cbf20.uploadid, then
-		//          scanDir is /mnt/mys3/mybucket/2012/photos/gophercon.png
-		// Case #3: if markerPath is /mnt/mys3/mybucket/2012/photos/, then
-		//          scanDir is /mnt/mys3/mybucket/2012/photos
-
-		scanDir = filepath.Dir(markerPath)
-	} else {
-		markerPath = bucketDir
-	}
-
-	// Have bucketDir ends with os.PathSeparator
-	if !strings.HasSuffix(bucketDir, string(os.PathSeparator)) {
-		bucketDir += string(os.PathSeparator)
-	}
-
-	// Remove os.PathSeparator if scanDir ends with
-	if strings.HasSuffix(scanDir, string(os.PathSeparator)) {
-		scanDir = filepath.Dir(scanDir)
-	}
-
-	// goroutine - retrieves directory entries, makes ObjectInfo and sends into the channel.
-	go func() {
-		defer close(objectInfoCh)
-
-		// send function - returns true if ObjectInfo is sent
-		// within (time.Second * 15) else false on timeout.
-		send := func(oi multipartObjectInfo) bool {
-			timer := time.After(time.Second * 15)
-			select {
-			case objectInfoCh <- oi:
-				return true
-			case <-timer:
-				return false
-			}
-		}
-
-		// filter function - filters directory entries matching multipart uploadids, prefix and marker
-		direntFilterFn := func(dirent fsDirent) bool {
-			// check if dirent is a directory (or) dirent is a regular file and it's name ends with Upload ID suffix string
-			if dirent.IsDir() || (dirent.IsRegular() && strings.HasSuffix(dirent.name, multipartUploadIDSuffix)) {
-				// return if dirent's name starts with prefixPath and lexically higher than markerPath
-				return strings.HasPrefix(dirent.name, prefixPath) && dirent.name > markerPath
-			}
-			return false
-		}
-
-		// filter function - filters directory entries matching multipart uploadids
-		subDirentFilterFn := func(dirent fsDirent) bool {
-			// check if dirent is a directory (or) dirent is a regular file and it's name ends with Upload ID suffix string
-			return dirent.IsDir() || (dirent.IsRegular() && strings.HasSuffix(dirent.name, multipartUploadIDSuffix))
-		}
-
-		// lastObjInfo is used to save last object info which is sent at last with End=true
-		var lastObjInfo *multipartObjectInfo
-
-		sendError := func(err error) {
-			if lastObjInfo != nil {
-				if !send(*lastObjInfo) {
-					// as we got error sending lastObjInfo, we can't send the error
-					return
-				}
-			}
-
-			send(multipartObjectInfo{Err: err, End: true})
-			return
-		}
-
-		for {
-			dirents, err := scandir(scanDir, direntFilterFn, false)
-			if err != nil {
-				sendError(err)
-				return
-			}
-
-			var dirent fsDirent
-			for len(dirents) > 0 {
-				dirent, dirents = dirents[0], dirents[1:]
-				if dirent.IsRegular() {
-					// Handle uploadid file
-					name := strings.Replace(filepath.Dir(dirent.name), bucketDir, "", 1)
-					if name == "" {
-						// This should not happen ie uploadid file should not be in bucket directory
-						sendError(errors.New("Corrupted metadata"))
-						return
-					}
-
-					uploadID := strings.Split(filepath.Base(dirent.name), multipartUploadIDSuffix)[0]
-
-					// Solaris and older unixes have modTime to be
-					// empty, fallback to os.Stat() to fill missing values.
-					if dirent.modTime.IsZero() {
-						if fi, e := os.Stat(dirent.name); e == nil {
-							dirent.modTime = fi.ModTime()
-						} else {
-							sendError(e)
-							return
-						}
-					}
-
-					objInfo := multipartObjectInfo{
-						Name:         name,
-						UploadID:     uploadID,
-						ModifiedTime: dirent.modTime,
-					}
-
-					// as we got new object info, send last object info and keep new object info as last object info
-					if lastObjInfo != nil {
-						if !send(*lastObjInfo) {
-							return
-						}
-					}
-					lastObjInfo = &objInfo
-
-					continue
-				}
-
-				// Fetch sub dirents.
-				subDirents, err := scandir(dirent.name, subDirentFilterFn, false)
-				if err != nil {
-					sendError(err)
-					return
-				}
-
-				subDirFound := false
-				uploadIDDirents := []fsDirent{}
-				// If subDirents has a directory, then current dirent needs to be sent
-				for _, subdirent := range subDirents {
-					if subdirent.IsDir() {
-						subDirFound = true
-
-						if recursive {
-							break
-						}
-					}
-
-					if !recursive && subdirent.IsRegular() {
-						uploadIDDirents = append(uploadIDDirents, subdirent)
-					}
-				}
-
-				// Send directory only for non-recursive listing
-				if !recursive && (subDirFound || len(subDirents) == 0) {
-					// Solaris and older unixes have modTime to be
-					// empty, fallback to os.Stat() to fill missing values.
-					if dirent.modTime.IsZero() {
-						if fi, e := os.Stat(dirent.name); e == nil {
-							dirent.modTime = fi.ModTime()
-						} else {
-							sendError(e)
-							return
-						}
-					}
-
-					objInfo := multipartObjectInfo{
-						Name:         strings.Replace(dirent.name, bucketDir, "", 1),
-						ModifiedTime: dirent.modTime,
-						IsDir:        true,
-					}
-
-					// as we got new object info, send last object info and keep new object info as last object info
-					if lastObjInfo != nil {
-						if !send(*lastObjInfo) {
-							return
-						}
-					}
-					lastObjInfo = &objInfo
-				}
-
-				if recursive {
-					dirents = append(subDirents, dirents...)
-				} else {
-					dirents = append(uploadIDDirents, dirents...)
-				}
-			}
-
-			if !recursive {
-				break
-			}
-
-			markerPath = scanDir + string(os.PathSeparator)
-			if scanDir = filepath.Dir(scanDir); scanDir < dirDepth {
-				break
-			}
-		}
-
-		if lastObjInfo != nil {
-			// we got last object
-			lastObjInfo.End = true
-			if !send(*lastObjInfo) {
-				return
-			}
-		}
-	}()
-
-	return objectInfoCh
-}
-
-// multipartObjectInfo - Multipart object info
-type multipartObjectInfo struct {
-	Name         string
-	UploadID     string
-	ModifiedTime time.Time
-	IsDir        bool
-	Err          error
-	End          bool
-}

fs-multipart.go

@@ -1,685 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015,2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"crypto/md5"
-	"encoding/hex"
-	"errors"
-	"fmt"
-	"io"
-	"io/ioutil"
-	"os"
-	"path/filepath"
-	"strconv"
-	"strings"
-
-	"github.com/minio/minio/pkg/mimedb"
-	"github.com/minio/minio/pkg/probe"
-	"github.com/minio/minio/pkg/safe"
-	"github.com/skyrings/skyring-common/tools/uuid"
-)
-
-const (
-	minioMetaDir            = ".minio"
-	multipartUploadIDSuffix = ".uploadid"
-)
-
-// Removes files and its parent directories up to a given level.
-func removeFileTree(fileName string, level string) error {
-	if e := os.Remove(fileName); e != nil {
-		return e
-	}
-
-	for fileDir := filepath.Dir(fileName); fileDir > level; fileDir = filepath.Dir(fileDir) {
-		if status, e := isDirEmpty(fileDir); e != nil {
-			return e
-		} else if !status {
-			break
-		}
-		if e := os.Remove(fileDir); e != nil {
-			return e
-		}
-	}
-
-	return nil
-}
-
-// Takes an input stream and safely writes to disk, additionally
-// verifies checksum.
-func safeWriteFile(fileName string, data io.Reader, size int64, md5sum string) error {
-	safeFile, e := safe.CreateFileWithSuffix(fileName, "-")
-	if e != nil {
-		return e
-	}
-
-	md5Hasher := md5.New()
-	multiWriter := io.MultiWriter(md5Hasher, safeFile)
-	if size > 0 {
-		if _, e = io.CopyN(multiWriter, data, size); e != nil {
-			// Closes the file safely and removes it in a single atomic operation.
-			safeFile.CloseAndRemove()
-			return e
-		}
-	} else {
-		if _, e = io.Copy(multiWriter, data); e != nil {
-			// Closes the file safely and removes it in a single atomic operation.
-			safeFile.CloseAndRemove()
-			return e
-		}
-	}
-
-	dataMd5sum := hex.EncodeToString(md5Hasher.Sum(nil))
-	if md5sum != "" && !isMD5SumEqual(md5sum, dataMd5sum) {
-		// Closes the file safely and removes it in a single atomic operation.
-		safeFile.CloseAndRemove()
-		return BadDigest{ExpectedMD5: md5sum, CalculatedMD5: dataMd5sum}
-	}
-
-	// Safely close the file and atomically renames it the actual filePath.
-	safeFile.Close()
-
-	// Safely wrote the file.
-	return nil
-}
-
-func isFileExist(filename string) (bool, error) {
-	fi, e := os.Lstat(filename)
-	if e != nil {
-		if os.IsNotExist(e) {
-			return false, nil
-		}
-
-		return false, e
-	}
-
-	return fi.Mode().IsRegular(), nil
-}
-
-// Create an s3 compatible MD5sum for complete multipart transaction.
-func makeS3MD5(md5Strs ...string) (string, *probe.Error) {
-	var finalMD5Bytes []byte
-	for _, md5Str := range md5Strs {
-		md5Bytes, e := hex.DecodeString(md5Str)
-		if e != nil {
-			return "", probe.NewError(e)
-		}
-		finalMD5Bytes = append(finalMD5Bytes, md5Bytes...)
-	}
-	md5Hasher := md5.New()
-	md5Hasher.Write(finalMD5Bytes)
-	s3MD5 := fmt.Sprintf("%s-%d", hex.EncodeToString(md5Hasher.Sum(nil)), len(md5Strs))
-	return s3MD5, nil
-}
-
-func (fs Filesystem) newUploadID(bucket, object string) (string, error) {
-	metaObjectDir := filepath.Join(fs.diskPath, minioMetaDir, bucket, object)
-
-	// create metaObjectDir if not exist
-	if status, e := isDirExist(metaObjectDir); e != nil {
-		return "", e
-	} else if !status {
-		if e := os.MkdirAll(metaObjectDir, 0755); e != nil {
-			return "", e
-		}
-	}
-
-	for {
-		uuid, e := uuid.New()
-		if e != nil {
-			return "", e
-		}
-
-		uploadID := uuid.String()
-		uploadIDFile := filepath.Join(metaObjectDir, uploadID+multipartUploadIDSuffix)
-		if _, e := os.Lstat(uploadIDFile); e != nil {
-			if !os.IsNotExist(e) {
-				return "", e
-			}
-
-			// uploadIDFile doesn't exist, so create empty file to reserve the name
-			if e := ioutil.WriteFile(uploadIDFile, []byte{}, 0644); e != nil {
-				return "", e
-			}
-
-			return uploadID, nil
-		}
-		// uploadIDFile already exists.
-		// loop again to try with different uuid generated.
-	}
-}
-
-func (fs Filesystem) isUploadIDExist(bucket, object, uploadID string) (bool, error) {
-	return isFileExist(filepath.Join(fs.diskPath, minioMetaDir, bucket, object, uploadID+multipartUploadIDSuffix))
-}
-
-func (fs Filesystem) cleanupUploadID(bucket, object, uploadID string) error {
-	metaObjectDir := filepath.Join(fs.diskPath, minioMetaDir, bucket, object)
-	uploadIDPrefix := uploadID + "."
-
-	dirents, e := scandir(metaObjectDir,
-		func(dirent fsDirent) bool {
-			return dirent.IsRegular() && strings.HasPrefix(dirent.name, uploadIDPrefix)
-		},
-		true)
-
-	if e != nil {
-		return e
-	}
-
-	for _, dirent := range dirents {
-		if e := os.Remove(filepath.Join(metaObjectDir, dirent.name)); e != nil {
-			return e
-		}
-	}
-
-	if status, e := isDirEmpty(metaObjectDir); e != nil {
-		return e
-	} else if status {
-		if e := removeFileTree(metaObjectDir, filepath.Join(fs.diskPath, minioMetaDir, bucket)); e != nil {
-			return e
-		}
-	}
-
-	return nil
-}
-
-func (fs Filesystem) checkMultipartArgs(bucket, object string) (string, error) {
-	bucket, e := fs.checkBucketArg(bucket)
-	if e != nil {
-		return "", e
-	}
-
-	if !IsValidObjectName(object) {
-		return "", ObjectNameInvalid{Object: object}
-	}
-
-	return bucket, nil
-}
-
-// NewMultipartUpload - initiate a new multipart session
-func (fs Filesystem) NewMultipartUpload(bucket, object string) (string, *probe.Error) {
-	if bucketDirName, e := fs.checkMultipartArgs(bucket, object); e == nil {
-		bucket = bucketDirName
-	} else {
-		return "", probe.NewError(e)
-	}
-
-	if e := checkDiskFree(fs.diskPath, fs.minFreeDisk); e != nil {
-		return "", probe.NewError(e)
-	}
-
-	uploadID, e := fs.newUploadID(bucket, object)
-	if e != nil {
-		return "", probe.NewError(e)
-	}
-
-	return uploadID, nil
-}
-
-// PutObjectPart - create a part in a multipart session
-func (fs Filesystem) PutObjectPart(bucket, object, uploadID string, partNumber int, size int64, data io.Reader, md5Hex string) (string, *probe.Error) {
-	if bucketDirName, e := fs.checkMultipartArgs(bucket, object); e == nil {
-		bucket = bucketDirName
-	} else {
-		return "", probe.NewError(e)
-	}
-
-	if status, e := fs.isUploadIDExist(bucket, object, uploadID); e != nil {
-		//return "", probe.NewError(InternalError{Err: err})
-		return "", probe.NewError(e)
-	} else if !status {
-		return "", probe.NewError(InvalidUploadID{UploadID: uploadID})
-	}
-
-	// Part id cannot be negative.
-	if partNumber <= 0 {
-		return "", probe.NewError(errors.New("invalid part id, cannot be zero or less than zero"))
-	}
-
-	if partNumber > 10000 {
-		return "", probe.NewError(errors.New("invalid part id, should be not more than 10000"))
-	}
-
-	if e := checkDiskFree(fs.diskPath, fs.minFreeDisk); e != nil {
-		return "", probe.NewError(e)
-	}
-
-	partSuffix := fmt.Sprintf("%s.%d.%s", uploadID, partNumber, md5Hex)
-	partFilePath := filepath.Join(fs.diskPath, minioMetaDir, bucket, object, partSuffix)
-	if e := safeWriteFile(partFilePath, data, size, md5Hex); e != nil {
-		return "", probe.NewError(e)
-	}
-	return md5Hex, nil
-}
-
-// AbortMultipartUpload - abort an incomplete multipart session
-func (fs Filesystem) AbortMultipartUpload(bucket, object, uploadID string) *probe.Error {
-	if bucketDirName, e := fs.checkMultipartArgs(bucket, object); e == nil {
-		bucket = bucketDirName
-	} else {
-		return probe.NewError(e)
-	}
-
-	if status, e := fs.isUploadIDExist(bucket, object, uploadID); e != nil {
-		//return probe.NewError(InternalError{Err: err})
-		return probe.NewError(e)
-	} else if !status {
-		return probe.NewError(InvalidUploadID{UploadID: uploadID})
-	}
-
-	if e := fs.cleanupUploadID(bucket, object, uploadID); e != nil {
-		return probe.NewError(e)
-	}
-
-	return nil
-}
-
-// CompleteMultipartUpload - complete a multipart upload and persist the data
-func (fs Filesystem) CompleteMultipartUpload(bucket, object, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) {
-	if bucketDirName, e := fs.checkMultipartArgs(bucket, object); e == nil {
-		bucket = bucketDirName
-	} else {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	if status, e := fs.isUploadIDExist(bucket, object, uploadID); e != nil {
-		//return probe.NewError(InternalError{Err: err})
-		return ObjectInfo{}, probe.NewError(e)
-	} else if !status {
-		return ObjectInfo{}, probe.NewError(InvalidUploadID{UploadID: uploadID})
-	}
-
-	if e := checkDiskFree(fs.diskPath, fs.minFreeDisk); e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	metaObjectDir := filepath.Join(fs.diskPath, minioMetaDir, bucket, object)
-
-	var md5Sums []string
-	for _, part := range parts {
-		partNumber := part.PartNumber
-		md5sum := strings.Trim(part.ETag, "\"")
-		partFile := filepath.Join(metaObjectDir, uploadID+"."+strconv.Itoa(partNumber)+"."+md5sum)
-		if status, err := isFileExist(partFile); err != nil {
-			return ObjectInfo{}, probe.NewError(err)
-		} else if !status {
-			return ObjectInfo{}, probe.NewError(InvalidPart{})
-		}
-		md5Sums = append(md5Sums, md5sum)
-	}
-
-	// Save the s3 md5.
-	s3MD5, err := makeS3MD5(md5Sums...)
-	if err != nil {
-		return ObjectInfo{}, err.Trace(md5Sums...)
-	}
-
-	completeObjectFile := filepath.Join(metaObjectDir, uploadID+".complete.")
-	safeFile, e := safe.CreateFileWithSuffix(completeObjectFile, "-")
-	if e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-	for _, part := range parts {
-		partNumber := part.PartNumber
-		// Trim off the odd double quotes from ETag in the beginning and end.
-		md5sum := strings.TrimPrefix(part.ETag, "\"")
-		md5sum = strings.TrimSuffix(md5sum, "\"")
-		partFileStr := filepath.Join(metaObjectDir, fmt.Sprintf("%s.%d.%s", uploadID, partNumber, md5sum))
-		var partFile *os.File
-		partFile, e = os.Open(partFileStr)
-		if e != nil {
-			// Remove the complete file safely.
-			safeFile.CloseAndRemove()
-			return ObjectInfo{}, probe.NewError(e)
-		} else if _, e = io.Copy(safeFile, partFile); e != nil {
-			// Remove the complete file safely.
-			safeFile.CloseAndRemove()
-			return ObjectInfo{}, probe.NewError(e)
-		}
-		partFile.Close() // Close part file after successful copy.
-	}
-	// All parts concatenated, safely close the temp file.
-	safeFile.Close()
-
-	// Stat to gather fresh stat info.
-	objSt, e := os.Stat(completeObjectFile)
-	if e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	bucketPath := filepath.Join(fs.diskPath, bucket)
-	objectPath := filepath.Join(bucketPath, object)
-	if e = os.MkdirAll(filepath.Dir(objectPath), 0755); e != nil {
-		os.Remove(completeObjectFile)
-		return ObjectInfo{}, probe.NewError(e)
-	}
-	if e = os.Rename(completeObjectFile, objectPath); e != nil {
-		os.Remove(completeObjectFile)
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	fs.cleanupUploadID(bucket, object, uploadID) // TODO: handle and log the error
-
-	contentType := "application/octet-stream"
-	if objectExt := filepath.Ext(objectPath); objectExt != "" {
-		if content, ok := mimedb.DB[strings.ToLower(strings.TrimPrefix(objectExt, "."))]; ok {
-			contentType = content.ContentType
-		}
-	}
-
-	newObject := ObjectInfo{
-		Bucket:       bucket,
-		Name:         object,
-		ModifiedTime: objSt.ModTime(),
-		Size:         objSt.Size(),
-		ContentType:  contentType,
-		MD5Sum:       s3MD5,
-	}
-
-	return newObject, nil
-}
-
-func (fs *Filesystem) saveListMultipartObjectCh(params listMultipartObjectParams, ch <-chan multipartObjectInfo) {
-	fs.listMultipartObjectMapMutex.Lock()
-	defer fs.listMultipartObjectMapMutex.Unlock()
-
-	channels := []<-chan multipartObjectInfo{ch}
-	if _, ok := fs.listMultipartObjectMap[params]; ok {
-		channels = append(fs.listMultipartObjectMap[params], ch)
-	}
-
-	fs.listMultipartObjectMap[params] = channels
-}
-
-func (fs *Filesystem) lookupListMultipartObjectCh(params listMultipartObjectParams) <-chan multipartObjectInfo {
-	fs.listMultipartObjectMapMutex.Lock()
-	defer fs.listMultipartObjectMapMutex.Unlock()
-
-	if channels, ok := fs.listMultipartObjectMap[params]; ok {
-		var channel <-chan multipartObjectInfo
-		channel, channels = channels[0], channels[1:]
-		if len(channels) > 0 {
-			fs.listMultipartObjectMap[params] = channels
-		} else {
-			// do not store empty channel list
-			delete(fs.listMultipartObjectMap, params)
-		}
-
-		return channel
-	}
-
-	return nil
-}
-
-// ListMultipartUploads - list incomplete multipart sessions for a given BucketMultipartResourcesMetadata
-func (fs Filesystem) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) {
-	result := ListMultipartsInfo{}
-
-	bucket, e := fs.checkBucketArg(bucket)
-	if e != nil {
-		return result, probe.NewError(e)
-	}
-
-	if !IsValidObjectPrefix(objectPrefix) {
-		return result, probe.NewError(ObjectNameInvalid{Bucket: bucket, Object: objectPrefix})
-	}
-
-	prefixPath := filepath.FromSlash(objectPrefix)
-
-	// Verify if delimiter is anything other than '/', which we do not support.
-	if delimiter != "" && delimiter != "/" {
-		return result, probe.NewError(fmt.Errorf("delimiter '%s' is not supported", delimiter))
-	}
-
-	if keyMarker != "" && !strings.HasPrefix(keyMarker, objectPrefix) {
-		return result, probe.NewError(fmt.Errorf("Invalid combination of marker '%s' and prefix '%s'", keyMarker, objectPrefix))
-	}
-
-	markerPath := filepath.FromSlash(keyMarker)
-	if uploadIDMarker != "" {
-		if strings.HasSuffix(markerPath, string(os.PathSeparator)) {
-			return result, probe.NewError(fmt.Errorf("Invalid combination of uploadID marker '%s' and marker '%s'", uploadIDMarker, keyMarker))
-		}
-		id, e := uuid.Parse(uploadIDMarker)
-		if e != nil {
-			return result, probe.NewError(e)
-		}
-		if id.IsZero() {
-			return result, probe.NewError(fmt.Errorf("Invalid upload ID marker %s", uploadIDMarker))
-		}
-	}
-
-	// Return empty response if maxUploads is zero
-	if maxUploads == 0 {
-		return result, nil
-	}
-
-	// set listObjectsLimit to maxUploads for out-of-range limit
-	if maxUploads < 0 || maxUploads > listObjectsLimit {
-		maxUploads = listObjectsLimit
-	}
-
-	recursive := true
-	if delimiter == "/" {
-		recursive = false
-	}
-
-	metaBucketDir := filepath.Join(fs.diskPath, minioMetaDir, bucket)
-
-	// Lookup of if listMultipartObjectChannel is available for given
-	// parameters, else create a new one.
-	savedChannel := true
-	multipartObjectInfoCh := fs.lookupListMultipartObjectCh(listMultipartObjectParams{
-		bucket:         bucket,
-		delimiter:      delimiter,
-		keyMarker:      markerPath,
-		prefix:         prefixPath,
-		uploadIDMarker: uploadIDMarker,
-	})
-
-	if multipartObjectInfoCh == nil {
-		multipartObjectInfoCh = scanMultipartDir(metaBucketDir, objectPrefix, keyMarker, uploadIDMarker, recursive)
-		savedChannel = false
-	}
-
-	var objInfo *multipartObjectInfo
-	nextKeyMarker := ""
-	nextUploadIDMarker := ""
-	for i := 0; i < maxUploads; {
-		// read the channel
-		if oi, ok := <-multipartObjectInfoCh; ok {
-			objInfo = &oi
-		} else {
-			// closed channel
-			if i == 0 {
-				// first read
-				if !savedChannel {
-					// its valid to have a closed new channel for first read
-					multipartObjectInfoCh = nil
-					break
-				}
-
-				// invalid saved channel amd create new channel
-				multipartObjectInfoCh = scanMultipartDir(metaBucketDir, objectPrefix, keyMarker,
-					uploadIDMarker, recursive)
-			} else {
-				// TODO: FIX: there is a chance of infinite loop if we get closed channel always
-				// the channel got closed due to timeout
-				// create a new channel
-				multipartObjectInfoCh = scanMultipartDir(metaBucketDir, objectPrefix, nextKeyMarker,
-					nextUploadIDMarker, recursive)
-			}
-
-			// make it as new channel
-			savedChannel = false
-			continue
-		}
-
-		if objInfo.Err != nil {
-			if os.IsNotExist(objInfo.Err) {
-				return ListMultipartsInfo{}, nil
-			}
-
-			return ListMultipartsInfo{}, probe.NewError(objInfo.Err)
-		}
-
-		// backward compatibility check
-		if strings.Contains(objInfo.Name, "$multiparts") || strings.Contains(objInfo.Name, "$tmpobject") {
-			continue
-		}
-
-		// Directories are listed only if recursive is false
-		if objInfo.IsDir {
-			result.CommonPrefixes = append(result.CommonPrefixes, objInfo.Name)
-		} else {
-			result.Uploads = append(result.Uploads, uploadMetadata{
-				Object:    objInfo.Name,
-				UploadID:  objInfo.UploadID,
-				Initiated: objInfo.ModifiedTime,
-			})
-		}
-
-		nextKeyMarker = objInfo.Name
-		nextUploadIDMarker = objInfo.UploadID
-		i++
-
-		if objInfo.End {
-			// as we received last object, do not save this channel for later use
-			multipartObjectInfoCh = nil
-			break
-		}
-	}
-
-	if multipartObjectInfoCh != nil {
-		// we haven't received last object
-		result.IsTruncated = true
-		result.NextKeyMarker = nextKeyMarker
-		result.NextUploadIDMarker = nextUploadIDMarker
-
-		// save this channel for later use
-		fs.saveListMultipartObjectCh(listMultipartObjectParams{
-			bucket:         bucket,
-			delimiter:      delimiter,
-			keyMarker:      nextKeyMarker,
-			prefix:         objectPrefix,
-			uploadIDMarker: nextUploadIDMarker,
-		}, multipartObjectInfoCh)
-	}
-
-	return result, nil
-}
-
-// ListObjectParts - list parts from incomplete multipart session for a given ObjectResourcesMetadata
-func (fs Filesystem) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) {
-	if bucketDirName, err := fs.checkMultipartArgs(bucket, object); err == nil {
-		bucket = bucketDirName
-	} else {
-		return ListPartsInfo{}, probe.NewError(err)
-	}
-
-	if status, err := fs.isUploadIDExist(bucket, object, uploadID); err != nil {
-		//return probe.NewError(InternalError{Err: err})
-		return ListPartsInfo{}, probe.NewError(err)
-	} else if !status {
-		return ListPartsInfo{}, probe.NewError(InvalidUploadID{UploadID: uploadID})
-	}
-
-	// return empty ListPartsInfo
-	if maxParts == 0 {
-		return ListPartsInfo{}, nil
-	}
-
-	if maxParts < 0 || maxParts > 1000 {
-		maxParts = 1000
-	}
-
-	metaObjectDir := filepath.Join(fs.diskPath, minioMetaDir, bucket, object)
-	uploadIDPrefix := uploadID + "."
-
-	dirents, e := scandir(metaObjectDir,
-		func(dirent fsDirent) bool {
-			// Part file is a regular file and has to be started with 'UPLOADID.'
-			if !(dirent.IsRegular() && strings.HasPrefix(dirent.name, uploadIDPrefix)) {
-				return false
-			}
-
-			// Valid part file has to be 'UPLOADID.PARTNUMBER.MD5SUM'
-			tokens := strings.Split(dirent.name, ".")
-			if len(tokens) != 3 {
-				return false
-			}
-
-			if partNumber, err := strconv.Atoi(tokens[1]); err == nil {
-				if partNumber >= 1 && partNumber <= 10000 && partNumber > partNumberMarker {
-					return true
-				}
-			}
-
-			return false
-		},
-		true)
-	if e != nil {
-		return ListPartsInfo{}, probe.NewError(e)
-	}
-
-	isTruncated := false
-	nextPartNumberMarker := 0
-
-	parts := []partInfo{}
-	for i := range dirents {
-		if i == maxParts {
-			isTruncated = true
-			break
-		}
-
-		// In some OS modTime is empty and use os.Stat() to fill missing values
-		if dirents[i].modTime.IsZero() {
-			if fi, e := os.Stat(filepath.Join(metaObjectDir, dirents[i].name)); e == nil {
-				dirents[i].modTime = fi.ModTime()
-				dirents[i].size = fi.Size()
-			} else {
-				return ListPartsInfo{}, probe.NewError(e)
-			}
-		}
-
-		tokens := strings.Split(dirents[i].name, ".")
-		partNumber, _ := strconv.Atoi(tokens[1])
-		md5sum := tokens[2]
-		parts = append(parts, partInfo{
-			PartNumber:   partNumber,
-			LastModified: dirents[i].modTime,
-			ETag:         md5sum,
-			Size:         dirents[i].size,
-		})
-	}
-
-	if isTruncated {
-		nextPartNumberMarker = 0
-	}
-
-	return ListPartsInfo{
-		Bucket:               bucket,
-		Object:               object,
-		UploadID:             uploadID,
-		PartNumberMarker:     partNumberMarker,
-		NextPartNumberMarker: nextPartNumberMarker,
-		MaxParts:             maxParts,
-		IsTruncated:          isTruncated,
-		Parts:                parts,
-	}, nil
-}

fs-object.go

@@ -1,338 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"bytes"
-	"crypto/md5"
-	"io"
-	"os"
-	"path/filepath"
-	"strings"
-
-	"encoding/hex"
-	"runtime"
-
-	"github.com/minio/minio/pkg/mimedb"
-	"github.com/minio/minio/pkg/probe"
-	"github.com/minio/minio/pkg/safe"
-)
-
-/// Object Operations
-
-// GetObject - GET object
-func (fs Filesystem) GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error) {
-	// Input validation.
-	bucket, e := fs.checkBucketArg(bucket)
-	if e != nil {
-		return nil, probe.NewError(e)
-	}
-	if !IsValidObjectName(object) {
-		return nil, probe.NewError(ObjectNameInvalid{Bucket: bucket, Object: object})
-	}
-
-	objectPath := filepath.Join(fs.diskPath, bucket, object)
-	file, e := os.Open(objectPath)
-	if e != nil {
-		// If the object doesn't exist, the bucket might not exist either. Stat for
-		// the bucket and give a better error message if that is true.
-		if os.IsNotExist(e) {
-			_, e = os.Stat(filepath.Join(fs.diskPath, bucket))
-			if os.IsNotExist(e) {
-				return nil, probe.NewError(BucketNotFound{Bucket: bucket})
-			}
-			return nil, probe.NewError(ObjectNotFound{Bucket: bucket, Object: object})
-		}
-		return nil, probe.NewError(e)
-	}
-	// Initiate a cached stat operation on the file handler.
-	st, e := file.Stat()
-	if e != nil {
-		return nil, probe.NewError(e)
-	}
-	// Object path is a directory prefix, return object not found error.
-	if st.IsDir() {
-		return nil, probe.NewError(ObjectExistsAsPrefix{
-			Bucket: bucket,
-			Prefix: object,
-		})
-	}
-
-	// Seek to a starting offset.
-	_, e = file.Seek(startOffset, os.SEEK_SET)
-	if e != nil {
-		// When the "handle is invalid", the file might be a directory on Windows.
-		if runtime.GOOS == "windows" && strings.Contains(e.Error(), "handle is invalid") {
-			return nil, probe.NewError(ObjectNotFound{Bucket: bucket, Object: object})
-		}
-		return nil, probe.NewError(e)
-	}
-	return file, nil
-}
-
-// GetObjectInfo - get object info.
-func (fs Filesystem) GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error) {
-	// Input validation.
-	bucket, e := fs.checkBucketArg(bucket)
-	if e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	if !IsValidObjectName(object) {
-		return ObjectInfo{}, probe.NewError(ObjectNameInvalid{Bucket: bucket, Object: object})
-	}
-
-	info, err := getObjectInfo(fs.diskPath, bucket, object)
-	if err != nil {
-		if os.IsNotExist(err.ToGoError()) {
-			return ObjectInfo{}, probe.NewError(ObjectNotFound{Bucket: bucket, Object: object})
-		}
-		return ObjectInfo{}, err.Trace(bucket, object)
-	}
-	if info.IsDir {
-		return ObjectInfo{}, probe.NewError(ObjectNotFound{Bucket: bucket, Object: object})
-	}
-	return info, nil
-}
-
-// getObjectInfo - get object stat info.
-func getObjectInfo(rootPath, bucket, object string) (ObjectInfo, *probe.Error) {
-	// Do not use filepath.Join() since filepath.Join strips off any
-	// object names with '/', use them as is in a static manner so
-	// that we can send a proper 'ObjectNotFound' reply back upon os.Stat().
-	var objectPath string
-	// For windows use its special os.PathSeparator == "\\"
-	if runtime.GOOS == "windows" {
-		objectPath = rootPath + string(os.PathSeparator) + bucket + string(os.PathSeparator) + object
-	} else {
-		objectPath = rootPath + string(os.PathSeparator) + bucket + string(os.PathSeparator) + object
-	}
-	stat, e := os.Stat(objectPath)
-	if e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-	contentType := "application/octet-stream"
-	if runtime.GOOS == "windows" {
-		object = filepath.ToSlash(object)
-	}
-
-	if objectExt := filepath.Ext(object); objectExt != "" {
-		content, ok := mimedb.DB[strings.ToLower(strings.TrimPrefix(objectExt, "."))]
-		if ok {
-			contentType = content.ContentType
-		}
-	}
-	objInfo := ObjectInfo{
-		Bucket:       bucket,
-		Name:         object,
-		ModifiedTime: stat.ModTime(),
-		Size:         stat.Size(),
-		ContentType:  contentType,
-		IsDir:        stat.Mode().IsDir(),
-	}
-	return objInfo, nil
-}
-
-// isMD5SumEqual - returns error if md5sum mismatches, success its `nil`
-func isMD5SumEqual(expectedMD5Sum, actualMD5Sum string) bool {
-	// Verify the md5sum.
-	if expectedMD5Sum != "" && actualMD5Sum != "" {
-		// Decode md5sum to bytes from their hexadecimal
-		// representations.
-		expectedMD5SumBytes, err := hex.DecodeString(expectedMD5Sum)
-		if err != nil {
-			return false
-		}
-		actualMD5SumBytes, err := hex.DecodeString(actualMD5Sum)
-		if err != nil {
-			return false
-		}
-		// Verify md5sum bytes are equal after successful decoding.
-		if !bytes.Equal(expectedMD5SumBytes, actualMD5SumBytes) {
-			return false
-		}
-		return true
-	}
-	return false
-}
-
-// PutObject - create an object.
-func (fs Filesystem) PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error) {
-	// Check bucket name valid.
-	bucket, e := fs.checkBucketArg(bucket)
-	if e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	bucketDir := filepath.Join(fs.diskPath, bucket)
-
-	// Verify object path legal.
-	if !IsValidObjectName(object) {
-		return ObjectInfo{}, probe.NewError(ObjectNameInvalid{Bucket: bucket, Object: object})
-	}
-
-	if e = checkDiskFree(fs.diskPath, fs.minFreeDisk); e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	// Get object path.
-	objectPath := filepath.Join(bucketDir, object)
-
-	// md5Hex representation.
-	var md5Hex string
-	if len(metadata) != 0 {
-		md5Hex = metadata["md5Sum"]
-	}
-
-	// Write object.
-	safeFile, e := safe.CreateFileWithPrefix(objectPath, md5Hex+"$tmpobject")
-	if e != nil {
-		switch e := e.(type) {
-		case *os.PathError:
-			if e.Op == "mkdir" {
-				if strings.Contains(e.Error(), "not a directory") {
-					return ObjectInfo{}, probe.NewError(ObjectExistsAsPrefix{Bucket: bucket, Prefix: object})
-				}
-			}
-			return ObjectInfo{}, probe.NewError(e)
-		default:
-			return ObjectInfo{}, probe.NewError(e)
-		}
-	}
-
-	// Initialize md5 writer.
-	md5Writer := md5.New()
-
-	// Instantiate a new multi writer.
-	multiWriter := io.MultiWriter(md5Writer, safeFile)
-
-	// Instantiate checksum hashers and create a multiwriter.
-	if size > 0 {
-		if _, e = io.CopyN(multiWriter, data, size); e != nil {
-			safeFile.CloseAndRemove()
-			return ObjectInfo{}, probe.NewError(e)
-		}
-	} else {
-		if _, e = io.Copy(multiWriter, data); e != nil {
-			safeFile.CloseAndRemove()
-			return ObjectInfo{}, probe.NewError(e)
-		}
-	}
-
-	newMD5Hex := hex.EncodeToString(md5Writer.Sum(nil))
-	if md5Hex != "" {
-		if newMD5Hex != md5Hex {
-			return ObjectInfo{}, probe.NewError(BadDigest{md5Hex, newMD5Hex})
-		}
-	}
-
-	// Set stat again to get the latest metadata.
-	st, e := os.Stat(safeFile.Name())
-	if e != nil {
-		return ObjectInfo{}, probe.NewError(e)
-	}
-
-	contentType := "application/octet-stream"
-	if objectExt := filepath.Ext(objectPath); objectExt != "" {
-		content, ok := mimedb.DB[strings.ToLower(strings.TrimPrefix(objectExt, "."))]
-		if ok {
-			contentType = content.ContentType
-		}
-	}
-	newObject := ObjectInfo{
-		Bucket:       bucket,
-		Name:         object,
-		ModifiedTime: st.ModTime(),
-		Size:         st.Size(),
-		MD5Sum:       newMD5Hex,
-		ContentType:  contentType,
-	}
-
-	// Safely close and atomically rename the file.
-	safeFile.Close()
-
-	return newObject, nil
-}
-
-// deleteObjectPath - delete object path if its empty.
-func deleteObjectPath(basePath, deletePath, bucket, object string) *probe.Error {
-	if basePath == deletePath {
-		return nil
-	}
-	// Verify if the path exists.
-	pathSt, e := os.Stat(deletePath)
-	if e != nil {
-		if os.IsNotExist(e) {
-			return probe.NewError(ObjectNotFound{Bucket: bucket, Object: object})
-		}
-		return probe.NewError(e)
-	}
-	if pathSt.IsDir() {
-		// Verify if directory is empty.
-		empty, e := isDirEmpty(deletePath)
-		if e != nil {
-			return probe.NewError(e)
-		}
-		if !empty {
-			return nil
-		}
-	}
-	// Attempt to remove path.
-	if e := os.Remove(deletePath); e != nil {
-		return probe.NewError(e)
-	}
-	// Recursively go down the next path and delete again.
-	if err := deleteObjectPath(basePath, filepath.Dir(deletePath), bucket, object); err != nil {
-		return err.Trace(basePath, deletePath, bucket, object)
-	}
-	return nil
-}
-
-// DeleteObject - delete object.
-func (fs Filesystem) DeleteObject(bucket, object string) *probe.Error {
-	// Check bucket name valid
-	bucket, e := fs.checkBucketArg(bucket)
-	if e != nil {
-		return probe.NewError(e)
-	}
-
-	bucketDir := filepath.Join(fs.diskPath, bucket)
-
-	// Verify object path legal
-	if !IsValidObjectName(object) {
-		return probe.NewError(ObjectNameInvalid{Bucket: bucket, Object: object})
-	}
-
-	// Do not use filepath.Join() since filepath.Join strips off any
-	// object names with '/', use them as is in a static manner so
-	// that we can send a proper 'ObjectNotFound' reply back upon os.Stat().
-	var objectPath string
-	if runtime.GOOS == "windows" {
-		objectPath = fs.diskPath + string(os.PathSeparator) + bucket + string(os.PathSeparator) + object
-	} else {
-		objectPath = fs.diskPath + string(os.PathSeparator) + bucket + string(os.PathSeparator) + object
-	}
-	// Delete object path if its empty.
-	err := deleteObjectPath(bucketDir, objectPath, bucket, object)
-	if err != nil {
-		if os.IsNotExist(err.ToGoError()) {
-			return probe.NewError(ObjectNotFound{Bucket: bucket, Object: object})
-		}
-		return err.Trace(bucketDir, objectPath, bucket, object)
-	}
-	return nil
-}

fs-object_test.go

@@ -1,255 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"bytes"
-	"crypto/md5"
-	"encoding/hex"
-	"fmt"
-	"io"
-	"io/ioutil"
-	"os"
-	"path/filepath"
-	"strconv"
-	"strings"
-	"testing"
-)
-
-// Testing GetObjectInfo().
-func TestGetObjectInfo(t *testing.T) {
-	directory, e := ioutil.TempDir("", "minio-get-objinfo-test")
-	if e != nil {
-		t.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		t.Fatal(err)
-	}
-	// This bucket is used for testing getObjectInfo operations.
-	err = fs.MakeBucket("test-getobjectinfo")
-	if err != nil {
-		t.Fatal(err)
-	}
-	_, err = fs.PutObject("test-getobjectinfo", "Asia/asiapics.jpg", int64(len("asiapics")), bytes.NewBufferString("asiapics"), nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-	resultCases := []ObjectInfo{
-		// ObjectInfo -1.
-		// ObjectName set to a existing object in the test case (Test case 14).
-		{Bucket: "test-getobjectinfo", Name: "Asia/asiapics.jpg", ContentType: "image/jpeg", IsDir: false},
-	}
-	testCases := []struct {
-		bucketName string
-		objectName string
-
-		// Expected output of GetObjectInfo.
-		result ObjectInfo
-		err    error
-		// Flag indicating whether the test is expected to pass or not.
-		shouldPass bool
-	}{
-		// Test cases with invalid bucket names ( Test number 1-4 ).
-		{".test", "", ObjectInfo{}, BucketNameInvalid{Bucket: ".test"}, false},
-		{"Test", "", ObjectInfo{}, BucketNameInvalid{Bucket: "Test"}, false},
-		{"---", "", ObjectInfo{}, BucketNameInvalid{Bucket: "---"}, false},
-		{"ad", "", ObjectInfo{}, BucketNameInvalid{Bucket: "ad"}, false},
-		// Test cases with valid but non-existing bucket names (Test number 5-7).
-		{"abcdefgh", "abc", ObjectInfo{}, BucketNotFound{Bucket: "abcdefgh"}, false},
-		{"ijklmnop", "efg", ObjectInfo{}, BucketNotFound{Bucket: "ijklmnop"}, false},
-		// Test cases with valid but non-existing bucket names and invalid object name (Test number 8-9).
-		{"test-getobjectinfo", "", ObjectInfo{}, ObjectNameInvalid{Bucket: "test-getobjectinfo", Object: ""}, false},
-		{"test-getobjectinfo", "", ObjectInfo{}, ObjectNameInvalid{Bucket: "test-getobjectinfo", Object: ""}, false},
-		// Test cases with non-existing object name with existing bucket (Test number 10-12).
-		{"test-getobjectinfo", "Africa", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Africa"}, false},
-		{"test-getobjectinfo", "Antartica", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Antartica"}, false},
-		{"test-getobjectinfo", "Asia/myfile", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Asia/myfile"}, false},
-		// Test case with existing bucket but object name set to a directory (Test number 13).
-		{"test-getobjectinfo", "Asia", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Asia"}, false},
-		// Valid case with existing object (Test number 14).
-		{"test-getobjectinfo", "Asia/asiapics.jpg", resultCases[0], nil, true},
-	}
-	for i, testCase := range testCases {
-		result, err := fs.GetObjectInfo(testCase.bucketName, testCase.objectName)
-		if err != nil && testCase.shouldPass {
-			t.Errorf("Test %d: Expected to pass, but failed with: <ERROR> %s", i+1, err.Cause.Error())
-		}
-		if err == nil && !testCase.shouldPass {
-			t.Errorf("Test %d: Expected to fail with <ERROR> \"%s\", but passed instead", i+1, testCase.err.Error())
-		}
-		// Failed as expected, but does it fail for the expected reason.
-		if err != nil && !testCase.shouldPass {
-			if testCase.err.Error() != err.Cause.Error() {
-				t.Errorf("Test %d: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead", i+1, testCase.err.Error(), err.Cause.Error())
-			}
-		}
-
-		// Test passes as expected, but the output values are verified for correctness here.
-		if err == nil && testCase.shouldPass {
-			if testCase.result.Bucket != result.Bucket {
-				t.Fatalf("Test %d: Expected Bucket name to be '%s', but found '%s' instead", i+1, testCase.result.Bucket, result.Bucket)
-			}
-			if testCase.result.Name != result.Name {
-				t.Errorf("Test %d: Expected Object name to be %s, but instead found it to be %s", i+1, testCase.result.Name, result.Name)
-			}
-			if testCase.result.ContentType != result.ContentType {
-				t.Errorf("Test %d: Expected Content Type of the object to be %v, but instead found it to be %v", i+1, testCase.result.ContentType, result.ContentType)
-			}
-			if testCase.result.IsDir != result.IsDir {
-				t.Errorf("Test %d: Expected IsDir flag of the object to be %v, but instead found it to be %v", i+1, testCase.result.IsDir, result.IsDir)
-			}
-		}
-	}
-}
-
-// Testing getObjectInfo().
-func TestGetObjectInfoCore(t *testing.T) {
-	directory, e := ioutil.TempDir("", "minio-get-objinfo-test")
-	if e != nil {
-		t.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		t.Fatal(err)
-	}
-	// This bucket is used for testing getObjectInfo operations.
-	err = fs.MakeBucket("test-getobjinfo")
-	if err != nil {
-		t.Fatal(err)
-	}
-	_, err = fs.PutObject("test-getobjinfo", "Asia/asiapics.jpg", int64(len("asiapics")), bytes.NewBufferString("asiapics"), nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-	resultCases := []ObjectInfo{
-		// ObjectInfo - 1.
-		// ObjectName object name set to a existing directory in the test case.
-		{Bucket: "test-getobjinfo", Name: "Asia", Size: 0, ContentType: "application/octet-stream", IsDir: true},
-		// ObjectInfo -2.
-		// ObjectName set to a existing object in the test case.
-		{Bucket: "test-getobjinfo", Name: "Asia/asiapics.jpg", Size: int64(len("asiapics")), ContentType: "image/jpeg", IsDir: false},
-		// ObjectInfo-3.
-		// Object name set to a non-existing object in the test case.
-		{Bucket: "test-getobjinfo", Name: "Africa", Size: 0, ContentType: "image/jpeg", IsDir: false},
-	}
-	testCases := []struct {
-		bucketName string
-		objectName string
-
-		// Expected output of getObjectInfo.
-		result ObjectInfo
-		err    error
-
-		// Flag indicating whether the test is expected to pass or not.
-		shouldPass bool
-	}{
-		// Testcase with object name set to a existing directory ( Test number 1).
-		{"test-getobjinfo", "Asia", resultCases[0], nil, true},
-		// ObjectName set to a existing object ( Test number 2).
-		{"test-getobjinfo", "Asia/asiapics.jpg", resultCases[1], nil, true},
-		// Object name set to a non-existing object. (Test number 3).
-		{"test-getobjinfo", "Africa", resultCases[2], fmt.Errorf("%s", filepath.FromSlash("test-getobjinfo/Africa")), false},
-	}
-	rootPath := fs.(*Filesystem).GetRootPath()
-	for i, testCase := range testCases {
-		result, err := getObjectInfo(rootPath, testCase.bucketName, testCase.objectName)
-		if err != nil && testCase.shouldPass {
-			t.Errorf("Test %d: Expected to pass, but failed with: <ERROR> %s", i+1, err.Cause.Error())
-		}
-		if err == nil && !testCase.shouldPass {
-			t.Errorf("Test %d: Expected to fail with <ERROR> \"%s\", but passed instead", i+1, testCase.err.Error())
-		}
-		// Failed as expected, but does it fail for the expected reason.
-		if err != nil && !testCase.shouldPass {
-			if !strings.Contains(err.Cause.Error(), testCase.err.Error()) {
-				t.Errorf("Test %d: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead", i+1, testCase.err.Error(), err.Cause.Error())
-			}
-		}
-
-		// Test passes as expected, but the output values are verified for correctness here.
-		if err == nil && testCase.shouldPass {
-			if testCase.result.Bucket != result.Bucket {
-				t.Fatalf("Test %d: Expected Bucket name to be '%s', but found '%s' instead", i+1, testCase.result.Bucket, result.Bucket)
-			}
-			if testCase.result.Name != result.Name {
-				t.Errorf("Test %d: Expected Object name to be %s, but instead found it to be %s", i+1, testCase.result.Name, result.Name)
-			}
-			if testCase.result.ContentType != result.ContentType {
-				t.Errorf("Test %d: Expected Content Type of the object to be %v, but instead found it to be %v", i+1, testCase.result.ContentType, result.ContentType)
-			}
-			if testCase.result.IsDir != result.IsDir {
-				t.Errorf("Test %d: Expected IsDir flag of the object to be %v, but instead found it to be %v", i+1, testCase.result.IsDir, result.IsDir)
-			}
-		}
-	}
-}
-
-func BenchmarkGetObject(b *testing.B) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-benchmark-getobject")
-	if e != nil {
-		b.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
-
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		b.Fatal(err)
-	}
-
-	// Make a bucket and put in a few objects.
-	err = fs.MakeBucket("bucket")
-	if err != nil {
-		b.Fatal(err)
-	}
-
-	text := "Jack and Jill went up the hill / To fetch a pail of water."
-	hasher := md5.New()
-	hasher.Write([]byte(text))
-	metadata := make(map[string]string)
-	for i := 0; i < 10; i++ {
-		metadata["md5Sum"] = hex.EncodeToString(hasher.Sum(nil))
-		_, err = fs.PutObject("bucket", "object"+strconv.Itoa(i), int64(len(text)), bytes.NewBufferString(text), metadata)
-		if err != nil {
-			b.Fatal(err)
-		}
-	}
-
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		var buffer = new(bytes.Buffer)
-		r, err := fs.GetObject("bucket", "object"+strconv.Itoa(i%10), 0)
-		if err != nil {
-			b.Error(err)
-		}
-		if _, e := io.Copy(buffer, r); e != nil {
-			b.Error(e)
-		}
-		if buffer.Len() != len(text) {
-			b.Errorf("GetObject returned incorrect length %d (should be %d)\n", buffer.Len(), len(text))
-		}
-		r.Close()
-	}
-}

fs-utils.go

@@ -1,65 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"regexp"
-	"unicode/utf8"
-)
-
-// validBucket regexp.
-var validBucket = regexp.MustCompile(`^[a-z0-9][a-z0-9\.\-]{1,61}[a-z0-9]$`)
-
-// IsValidBucketName verifies a bucket name in accordance with Amazon's
-// requirements. It must be 3-63 characters long, can contain dashes
-// and periods, but must begin and end with a lowercase letter or a number.
-// See: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
-func IsValidBucketName(bucket string) bool {
-	if len(bucket) < 3 || len(bucket) > 63 {
-		return false
-	}
-	if bucket[0] == '.' || bucket[len(bucket)-1] == '.' {
-		return false
-	}
-	return validBucket.MatchString(bucket)
-}
-
-// IsValidObjectName verifies an object name in accordance with Amazon's
-// requirements. It cannot exceed 1024 characters and must be a valid UTF8
-// string.
-// See: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
-func IsValidObjectName(object string) bool {
-	if len(object) > 1024 || len(object) == 0 {
-		return false
-	}
-	if !utf8.ValidString(object) {
-		return false
-	}
-	return true
-}
-
-// IsValidObjectPrefix verifies whether the prefix is a valid object name.
-// Its valid to have a empty prefix.
-func IsValidObjectPrefix(object string) bool {
-	// Prefix can be empty.
-	if object == "" {
-		return true
-	}
-	// Verify if prefix is a valid object name.
-	return IsValidObjectName(object)
-
-}

fs-v1-metadata.go

@@ -0,0 +1,95 @@
+package main
+
+import (
+	"encoding/json"
+	"path"
+	"sort"
+)
+
+const (
+	fsMetaJSONFile = "fs.json"
+)
+
+// A fsMetaV1 represents a metadata header mapping keys to sets of values.
+type fsMetaV1 struct {
+	Version string `json:"version"`
+	Format  string `json:"format"`
+	Minio   struct {
+		Release string `json:"release"`
+	} `json:"minio"`
+	Parts []objectPartInfo `json:"parts,omitempty"`
+}
+
+// ObjectPartIndex - returns the index of matching object part number.
+func (m fsMetaV1) ObjectPartIndex(partNumber int) (partIndex int) {
+	for i, part := range m.Parts {
+		if partNumber == part.Number {
+			partIndex = i
+			return partIndex
+		}
+	}
+	return -1
+}
+
+// AddObjectPart - add a new object part in order.
+func (m *fsMetaV1) AddObjectPart(partNumber int, partName string, partETag string, partSize int64) {
+	partInfo := objectPartInfo{
+		Number: partNumber,
+		Name:   partName,
+		ETag:   partETag,
+		Size:   partSize,
+	}
+
+	// Update part info if it already exists.
+	for i, part := range m.Parts {
+		if partNumber == part.Number {
+			m.Parts[i] = partInfo
+			return
+		}
+	}
+
+	// Proceed to include new part info.
+	m.Parts = append(m.Parts, partInfo)
+
+	// Parts in fsMeta should be in sorted order by part number.
+	sort.Sort(byObjectPartNumber(m.Parts))
+}
+
+// readFSMetadata - returns the object metadata `fs.json` content.
+func (fs fsObjects) readFSMetadata(bucket, object string) (fsMeta fsMetaV1, err error) {
+	var buffer []byte
+	buffer, err = readAll(fs.storage, bucket, path.Join(object, fsMetaJSONFile))
+	if err != nil {
+		return fsMetaV1{}, err
+	}
+	err = json.Unmarshal(buffer, &fsMeta)
+	if err != nil {
+		return fsMetaV1{}, err
+	}
+	return fsMeta, nil
+}
+
+// newFSMetaV1 - initializes new fsMetaV1.
+func newFSMetaV1() (fsMeta fsMetaV1) {
+	fsMeta = fsMetaV1{}
+	fsMeta.Version = "1"
+	fsMeta.Format = "fs"
+	fsMeta.Minio.Release = minioReleaseTag
+	return fsMeta
+}
+
+// writeFSMetadata - writes `fs.json` metadata.
+func (fs fsObjects) writeFSMetadata(bucket, prefix string, fsMeta fsMetaV1) error {
+	metadataBytes, err := json.Marshal(fsMeta)
+	if err != nil {
+		return err
+	}
+	n, err := fs.storage.AppendFile(bucket, path.Join(prefix, fsMetaJSONFile), metadataBytes)
+	if err != nil {
+		return err
+	}
+	if n != int64(len(metadataBytes)) {
+		return errUnexpected
+	}
+	return nil
+}

fs-v1-multipart-common.go

@@ -0,0 +1,70 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"path"
+	"strings"
+)
+
+// Returns if the prefix is a multipart upload.
+func (fs fsObjects) isMultipartUpload(bucket, prefix string) bool {
+	_, err := fs.storage.StatFile(bucket, pathJoin(prefix, uploadsJSONFile))
+	return err == nil
+}
+
+// listUploadsInfo - list all uploads info.
+func (fs fsObjects) listUploadsInfo(prefixPath string) (uploads []uploadInfo, err error) {
+	splitPrefixes := strings.SplitN(prefixPath, "/", 3)
+	uploadIDs, err := readUploadsJSON(splitPrefixes[1], splitPrefixes[2], fs.storage)
+	if err != nil {
+		if err == errFileNotFound {
+			return []uploadInfo{}, nil
+		}
+		return nil, err
+	}
+	uploads = uploadIDs.Uploads
+	return uploads, nil
+}
+
+// Checks whether bucket exists.
+func (fs fsObjects) isBucketExist(bucket string) bool {
+	// Check whether bucket exists.
+	_, err := fs.storage.StatVol(bucket)
+	if err != nil {
+		if err == errVolumeNotFound {
+			return false
+		}
+		errorIf(err, "Stat failed on bucket "+bucket+".")
+		return false
+	}
+	return true
+}
+
+// isUploadIDExists - verify if a given uploadID exists and is valid.
+func (fs fsObjects) isUploadIDExists(bucket, object, uploadID string) bool {
+	uploadIDPath := path.Join(mpartMetaPrefix, bucket, object, uploadID)
+	_, err := fs.storage.StatFile(minioMetaBucket, path.Join(uploadIDPath, fsMetaJSONFile))
+	if err != nil {
+		if err == errFileNotFound {
+			return false
+		}
+		errorIf(err, "Unable to access upload id"+uploadIDPath)
+		return false
+	}
+	return true
+}

fs-v1-multipart.go

@@ -0,0 +1,648 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"crypto/md5"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"path"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/skyrings/skyring-common/tools/uuid"
+)
+
+// listMultipartUploads - lists all multipart uploads.
+func (fs fsObjects) listMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
+	result := ListMultipartsInfo{}
+	recursive := true
+	if delimiter == slashSeparator {
+		recursive = false
+	}
+
+	result.IsTruncated = true
+	result.MaxUploads = maxUploads
+	result.KeyMarker = keyMarker
+	result.Prefix = prefix
+	result.Delimiter = delimiter
+
+	// Not using path.Join() as it strips off the trailing '/'.
+	multipartPrefixPath := pathJoin(mpartMetaPrefix, bucket, prefix)
+	if prefix == "" {
+		// Should have a trailing "/" if prefix is ""
+		// For ex. multipartPrefixPath should be "multipart/bucket/" if prefix is ""
+		multipartPrefixPath += slashSeparator
+	}
+	multipartMarkerPath := ""
+	if keyMarker != "" {
+		multipartMarkerPath = pathJoin(mpartMetaPrefix, bucket, keyMarker)
+	}
+	var uploads []uploadMetadata
+	var err error
+	var eof bool
+	if uploadIDMarker != "" {
+		nsMutex.RLock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, keyMarker))
+		uploads, _, err = listMultipartUploadIDs(bucket, keyMarker, uploadIDMarker, maxUploads, fs.storage)
+		nsMutex.RUnlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, keyMarker))
+		if err != nil {
+			return ListMultipartsInfo{}, err
+		}
+		maxUploads = maxUploads - len(uploads)
+	}
+	if maxUploads > 0 {
+		walker := fs.lookupTreeWalk(listParams{minioMetaBucket, recursive, multipartMarkerPath, multipartPrefixPath})
+		if walker == nil {
+			walker = fs.startTreeWalk(minioMetaBucket, multipartPrefixPath, multipartMarkerPath, recursive, func(bucket, object string) bool {
+				return fs.isMultipartUpload(bucket, object)
+			})
+		}
+		for maxUploads > 0 {
+			walkResult, ok := <-walker.ch
+			if !ok {
+				// Closed channel.
+				eof = true
+				break
+			}
+			// For any walk error return right away.
+			if walkResult.err != nil {
+				// File not found or Disk not found is a valid case.
+				if walkResult.err == errFileNotFound || walkResult.err == errDiskNotFound {
+					eof = true
+					break
+				}
+				return ListMultipartsInfo{}, err
+			}
+			entry := strings.TrimPrefix(walkResult.entry, retainSlash(pathJoin(mpartMetaPrefix, bucket)))
+			if strings.HasSuffix(walkResult.entry, slashSeparator) {
+				uploads = append(uploads, uploadMetadata{
+					Object: entry,
+				})
+				maxUploads--
+				if maxUploads == 0 {
+					if walkResult.end {
+						eof = true
+						break
+					}
+				}
+				continue
+			}
+			var tmpUploads []uploadMetadata
+			var end bool
+			uploadIDMarker = ""
+			nsMutex.RLock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, entry))
+			tmpUploads, end, err = listMultipartUploadIDs(bucket, entry, uploadIDMarker, maxUploads, fs.storage)
+			nsMutex.RUnlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, entry))
+			if err != nil {
+				return ListMultipartsInfo{}, err
+			}
+			uploads = append(uploads, tmpUploads...)
+			maxUploads -= len(tmpUploads)
+			if walkResult.end && end {
+				eof = true
+				break
+			}
+		}
+	}
+	// Loop through all the received uploads fill in the multiparts result.
+	for _, upload := range uploads {
+		var objectName string
+		var uploadID string
+		if strings.HasSuffix(upload.Object, slashSeparator) {
+			// All directory entries are common prefixes.
+			uploadID = "" // Upload ids are empty for CommonPrefixes.
+			objectName = upload.Object
+			result.CommonPrefixes = append(result.CommonPrefixes, objectName)
+		} else {
+			uploadID = upload.UploadID
+			objectName = upload.Object
+			result.Uploads = append(result.Uploads, upload)
+		}
+		result.NextKeyMarker = objectName
+		result.NextUploadIDMarker = uploadID
+	}
+	result.IsTruncated = !eof
+	if !result.IsTruncated {
+		result.NextKeyMarker = ""
+		result.NextUploadIDMarker = ""
+	}
+	return result, nil
+}
+
+// ListMultipartUploads - lists all the pending multipart uploads on a
+// bucket. Additionally takes 'prefix, keyMarker, uploadIDmarker and a
+// delimiter' which allows us to list uploads match a particular
+// prefix or lexically starting from 'keyMarker' or delimiting the
+// output to get a directory like listing.
+//
+// Implements S3 compatible ListMultipartUploads API. The resulting
+// ListMultipartsInfo structure is unmarshalled directly into XML and
+// replied back to the client.
+func (fs fsObjects) ListMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, error) {
+	// Validate input arguments.
+	if !IsValidBucketName(bucket) {
+		return ListMultipartsInfo{}, BucketNameInvalid{Bucket: bucket}
+	}
+	if !fs.isBucketExist(bucket) {
+		return ListMultipartsInfo{}, BucketNotFound{Bucket: bucket}
+	}
+	if !IsValidObjectPrefix(prefix) {
+		return ListMultipartsInfo{}, ObjectNameInvalid{Bucket: bucket, Object: prefix}
+	}
+	// Verify if delimiter is anything other than '/', which we do not support.
+	if delimiter != "" && delimiter != slashSeparator {
+		return ListMultipartsInfo{}, UnsupportedDelimiter{
+			Delimiter: delimiter,
+		}
+	}
+	// Verify if marker has prefix.
+	if keyMarker != "" && !strings.HasPrefix(keyMarker, prefix) {
+		return ListMultipartsInfo{}, InvalidMarkerPrefixCombination{
+			Marker: keyMarker,
+			Prefix: prefix,
+		}
+	}
+	if uploadIDMarker != "" {
+		if strings.HasSuffix(keyMarker, slashSeparator) {
+			return ListMultipartsInfo{}, InvalidUploadIDKeyCombination{
+				UploadIDMarker: uploadIDMarker,
+				KeyMarker:      keyMarker,
+			}
+		}
+		id, err := uuid.Parse(uploadIDMarker)
+		if err != nil {
+			return ListMultipartsInfo{}, err
+		}
+		if id.IsZero() {
+			return ListMultipartsInfo{}, MalformedUploadID{
+				UploadID: uploadIDMarker,
+			}
+		}
+	}
+	return fs.listMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter, maxUploads)
+}
+
+// newMultipartUpload - wrapper for initializing a new multipart
+// request, returns back a unique upload id.
+//
+// Internally this function creates 'uploads.json' associated for the
+// incoming object at '.minio/multipart/bucket/object/uploads.json' on
+// all the disks. `uploads.json` carries metadata regarding on going
+// multipart operation on the object.
+func (fs fsObjects) newMultipartUpload(bucket string, object string, meta map[string]string) (uploadID string, err error) {
+	// No metadata is set, allocate a new one.
+	if meta == nil {
+		meta = make(map[string]string)
+	}
+
+	// Initialize `fs.json` values.
+	fsMeta := newFSMetaV1()
+
+	// This lock needs to be held for any changes to the directory contents of ".minio/multipart/object/"
+	nsMutex.Lock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object))
+	defer nsMutex.Unlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object))
+
+	uploadID = getUUID()
+	initiated := time.Now().UTC()
+	// Create 'uploads.json'
+	if err = writeUploadJSON(bucket, object, uploadID, initiated, fs.storage); err != nil {
+		return "", err
+	}
+	uploadIDPath := path.Join(mpartMetaPrefix, bucket, object, uploadID)
+	tempUploadIDPath := path.Join(tmpMetaPrefix, uploadID)
+	if err = fs.writeFSMetadata(minioMetaBucket, tempUploadIDPath, fsMeta); err != nil {
+		return "", toObjectErr(err, minioMetaBucket, tempUploadIDPath)
+	}
+	err = fs.storage.RenameFile(minioMetaBucket, path.Join(tempUploadIDPath, fsMetaJSONFile), minioMetaBucket, path.Join(uploadIDPath, fsMetaJSONFile))
+	if err != nil {
+		if dErr := fs.storage.DeleteFile(minioMetaBucket, path.Join(tempUploadIDPath, fsMetaJSONFile)); dErr != nil {
+			return "", toObjectErr(dErr, minioMetaBucket, tempUploadIDPath)
+		}
+		return "", toObjectErr(err, minioMetaBucket, uploadIDPath)
+	}
+	// Return success.
+	return uploadID, nil
+}
+
+// NewMultipartUpload - initialize a new multipart upload, returns a
+// unique id. The unique id returned here is of UUID form, for each
+// subsequent request each UUID is unique.
+//
+// Implements S3 compatible initiate multipart API.
+func (fs fsObjects) NewMultipartUpload(bucket, object string, meta map[string]string) (string, error) {
+	meta = make(map[string]string) // Reset the meta value, we are not going to save headers for fs.
+	// Verify if bucket name is valid.
+	if !IsValidBucketName(bucket) {
+		return "", BucketNameInvalid{Bucket: bucket}
+	}
+	// Verify whether the bucket exists.
+	if !fs.isBucketExist(bucket) {
+		return "", BucketNotFound{Bucket: bucket}
+	}
+	// Verify if object name is valid.
+	if !IsValidObjectName(object) {
+		return "", ObjectNameInvalid{Bucket: bucket, Object: object}
+	}
+	return fs.newMultipartUpload(bucket, object, meta)
+}
+
+// PutObjectPart - reads incoming data until EOF for the part file on
+// an ongoing multipart transaction. Internally incoming data is
+// written to '.minio/tmp' location and safely renamed to
+// '.minio/multipart' for reach parts.
+func (fs fsObjects) PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, error) {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return "", BucketNameInvalid{Bucket: bucket}
+	}
+	// Verify whether the bucket exists.
+	if !fs.isBucketExist(bucket) {
+		return "", BucketNotFound{Bucket: bucket}
+	}
+	if !IsValidObjectName(object) {
+		return "", ObjectNameInvalid{Bucket: bucket, Object: object}
+	}
+
+	uploadIDPath := path.Join(mpartMetaPrefix, bucket, object, uploadID)
+
+	nsMutex.RLock(minioMetaBucket, uploadIDPath)
+	// Just check if the uploadID exists to avoid copy if it doesn't.
+	uploadIDExists := fs.isUploadIDExists(bucket, object, uploadID)
+	nsMutex.RUnlock(minioMetaBucket, uploadIDPath)
+	if !uploadIDExists {
+		return "", InvalidUploadID{UploadID: uploadID}
+	}
+
+	// Hold write lock on the part so that there is no parallel upload on the part.
+	nsMutex.Lock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID, strconv.Itoa(partID)))
+	defer nsMutex.Unlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID, strconv.Itoa(partID)))
+
+	partSuffix := fmt.Sprintf("object%d", partID)
+	tmpPartPath := path.Join(tmpMetaPrefix, uploadID, partSuffix)
+
+	// Initialize md5 writer.
+	md5Writer := md5.New()
+
+	var buf = make([]byte, blockSizeV1)
+	for {
+		n, err := io.ReadFull(data, buf)
+		if err == io.EOF {
+			break
+		}
+		if err != nil && err != io.ErrUnexpectedEOF {
+			return "", toObjectErr(err, bucket, object)
+		}
+		// Update md5 writer.
+		md5Writer.Write(buf[:n])
+		m, err := fs.storage.AppendFile(minioMetaBucket, tmpPartPath, buf[:n])
+		if err != nil {
+			return "", toObjectErr(err, bucket, object)
+		}
+		if m != int64(len(buf[:n])) {
+			return "", toObjectErr(errUnexpected, bucket, object)
+		}
+	}
+
+	newMD5Hex := hex.EncodeToString(md5Writer.Sum(nil))
+	if md5Hex != "" {
+		if newMD5Hex != md5Hex {
+			return "", BadDigest{md5Hex, newMD5Hex}
+		}
+	}
+
+	// Hold write lock as we are updating fs.json
+	nsMutex.Lock(minioMetaBucket, uploadIDPath)
+	defer nsMutex.Unlock(minioMetaBucket, uploadIDPath)
+
+	// Just check if the uploadID exists to avoid copy if it doesn't.
+	if !fs.isUploadIDExists(bucket, object, uploadID) {
+		return "", InvalidUploadID{UploadID: uploadID}
+	}
+
+	fsMeta, err := fs.readFSMetadata(minioMetaBucket, uploadIDPath)
+	if err != nil {
+		return "", toObjectErr(err, minioMetaBucket, uploadIDPath)
+	}
+	fsMeta.AddObjectPart(partID, partSuffix, newMD5Hex, size)
+
+	partPath := path.Join(mpartMetaPrefix, bucket, object, uploadID, partSuffix)
+	err = fs.storage.RenameFile(minioMetaBucket, tmpPartPath, minioMetaBucket, partPath)
+	if err != nil {
+		if dErr := fs.storage.DeleteFile(minioMetaBucket, tmpPartPath); dErr != nil {
+			return "", toObjectErr(dErr, minioMetaBucket, tmpPartPath)
+		}
+		return "", toObjectErr(err, minioMetaBucket, partPath)
+	}
+	uploadIDPath = path.Join(mpartMetaPrefix, bucket, object, uploadID)
+	tempUploadIDPath := path.Join(tmpMetaPrefix, uploadID)
+	if err = fs.writeFSMetadata(minioMetaBucket, tempUploadIDPath, fsMeta); err != nil {
+		return "", toObjectErr(err, minioMetaBucket, tempUploadIDPath)
+	}
+	err = fs.storage.RenameFile(minioMetaBucket, path.Join(tempUploadIDPath, fsMetaJSONFile), minioMetaBucket, path.Join(uploadIDPath, fsMetaJSONFile))
+	if err != nil {
+		if dErr := fs.storage.DeleteFile(minioMetaBucket, path.Join(tempUploadIDPath, fsMetaJSONFile)); dErr != nil {
+			return "", toObjectErr(dErr, minioMetaBucket, tempUploadIDPath)
+		}
+		return "", toObjectErr(err, minioMetaBucket, uploadIDPath)
+	}
+	return newMD5Hex, nil
+}
+
+// listObjectParts - wrapper scanning through
+// '.minio/multipart/bucket/object/UPLOADID'. Lists all the parts
+// saved inside '.minio/multipart/bucket/object/UPLOADID'.
+func (fs fsObjects) listObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, error) {
+	result := ListPartsInfo{}
+
+	uploadIDPath := path.Join(mpartMetaPrefix, bucket, object, uploadID)
+	fsMeta, err := fs.readFSMetadata(minioMetaBucket, uploadIDPath)
+	if err != nil {
+		return ListPartsInfo{}, toObjectErr(err, minioMetaBucket, uploadIDPath)
+	}
+	// Only parts with higher part numbers will be listed.
+	partIdx := fsMeta.ObjectPartIndex(partNumberMarker)
+	parts := fsMeta.Parts
+	if partIdx != -1 {
+		parts = fsMeta.Parts[partIdx+1:]
+	}
+	count := maxParts
+	for _, part := range parts {
+		var fi FileInfo
+		partNamePath := path.Join(mpartMetaPrefix, bucket, object, uploadID, part.Name)
+		fi, err = fs.storage.StatFile(minioMetaBucket, partNamePath)
+		if err != nil {
+			return ListPartsInfo{}, toObjectErr(err, minioMetaBucket, partNamePath)
+		}
+		result.Parts = append(result.Parts, partInfo{
+			PartNumber:   part.Number,
+			ETag:         part.ETag,
+			LastModified: fi.ModTime,
+			Size:         fi.Size,
+		})
+		count--
+		if count == 0 {
+			break
+		}
+	}
+	// If listed entries are more than maxParts, we set IsTruncated as true.
+	if len(parts) > len(result.Parts) {
+		result.IsTruncated = true
+		// Make sure to fill next part number marker if IsTruncated is
+		// true for subsequent listing.
+		nextPartNumberMarker := result.Parts[len(result.Parts)-1].PartNumber
+		result.NextPartNumberMarker = nextPartNumberMarker
+	}
+	result.Bucket = bucket
+	result.Object = object
+	result.UploadID = uploadID
+	result.MaxParts = maxParts
+	return result, nil
+}
+
+// ListObjectParts - lists all previously uploaded parts for a given
+// object and uploadID.  Takes additional input of part-number-marker
+// to indicate where the listing should begin from.
+//
+// Implements S3 compatible ListObjectParts API. The resulting
+// ListPartsInfo structure is unmarshalled directly into XML and
+// replied back to the client.
+func (fs fsObjects) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, error) {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return ListPartsInfo{}, BucketNameInvalid{Bucket: bucket}
+	}
+	// Verify whether the bucket exists.
+	if !fs.isBucketExist(bucket) {
+		return ListPartsInfo{}, BucketNotFound{Bucket: bucket}
+	}
+	if !IsValidObjectName(object) {
+		return ListPartsInfo{}, ObjectNameInvalid{Bucket: bucket, Object: object}
+	}
+	// Hold lock so that there is no competing abort-multipart-upload or complete-multipart-upload.
+	nsMutex.Lock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID))
+	defer nsMutex.Unlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID))
+
+	if !fs.isUploadIDExists(bucket, object, uploadID) {
+		return ListPartsInfo{}, InvalidUploadID{UploadID: uploadID}
+	}
+	return fs.listObjectParts(bucket, object, uploadID, partNumberMarker, maxParts)
+}
+
+// CompleteMultipartUpload - completes an ongoing multipart
+// transaction after receiving all the parts indicated by the client.
+// Returns an md5sum calculated by concatenating all the individual
+// md5sums of all the parts.
+//
+// Implements S3 compatible Complete multipart API.
+func (fs fsObjects) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (string, error) {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return "", BucketNameInvalid{Bucket: bucket}
+	}
+	// Verify whether the bucket exists.
+	if !fs.isBucketExist(bucket) {
+		return "", BucketNotFound{Bucket: bucket}
+	}
+	if !IsValidObjectName(object) {
+		return "", ObjectNameInvalid{
+			Bucket: bucket,
+			Object: object,
+		}
+	}
+
+	uploadIDPath := path.Join(mpartMetaPrefix, bucket, object, uploadID)
+	// Hold lock so that
+	// 1) no one aborts this multipart upload
+	// 2) no one does a parallel complete-multipart-upload on this
+	// multipart upload
+	nsMutex.Lock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID))
+	defer nsMutex.Unlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID))
+
+	if !fs.isUploadIDExists(bucket, object, uploadID) {
+		return "", InvalidUploadID{UploadID: uploadID}
+	}
+
+	// Read saved fs metadata for ongoing multipart.
+	fsMeta, err := fs.readFSMetadata(minioMetaBucket, uploadIDPath)
+	if err != nil {
+		return "", toObjectErr(err, minioMetaBucket, uploadIDPath)
+	}
+
+	// Calculate s3 compatible md5sum for complete multipart.
+	s3MD5, err := completeMultipartMD5(parts...)
+	if err != nil {
+		return "", err
+	}
+
+	tempObj := path.Join(tmpMetaPrefix, uploadID, "object1")
+	var buffer = make([]byte, blockSizeV1)
+
+	// Loop through all parts, validate them and then commit to disk.
+	for i, part := range parts {
+		partIdx := fsMeta.ObjectPartIndex(part.PartNumber)
+		if partIdx == -1 {
+			return "", InvalidPart{}
+		}
+		if fsMeta.Parts[partIdx].ETag != part.ETag {
+			return "", BadDigest{}
+		}
+		// All parts except the last part has to be atleast 5MB.
+		if (i < len(parts)-1) && !isMinAllowedPartSize(fsMeta.Parts[partIdx].Size) {
+			return "", PartTooSmall{}
+		}
+		// Construct part suffix.
+		partSuffix := fmt.Sprintf("object%d", part.PartNumber)
+		multipartPartFile := path.Join(mpartMetaPrefix, bucket, object, uploadID, partSuffix)
+		offset := int64(0)
+		totalLeft := fsMeta.Parts[partIdx].Size
+		for totalLeft > 0 {
+			var n int64
+			n, err = fs.storage.ReadFile(minioMetaBucket, multipartPartFile, offset, buffer)
+			if err != nil {
+				if err == errFileNotFound {
+					return "", InvalidPart{}
+				}
+				return "", toObjectErr(err, minioMetaBucket, multipartPartFile)
+			}
+			n, err = fs.storage.AppendFile(minioMetaBucket, tempObj, buffer[:n])
+			if err != nil {
+				return "", toObjectErr(err, minioMetaBucket, tempObj)
+			}
+			offset += n
+			totalLeft -= n
+		}
+	}
+
+	// Rename the file back to original location, if not delete the temporary object.
+	err = fs.storage.RenameFile(minioMetaBucket, tempObj, bucket, object)
+	if err != nil {
+		if dErr := fs.storage.DeleteFile(minioMetaBucket, tempObj); dErr != nil {
+			return "", toObjectErr(dErr, minioMetaBucket, tempObj)
+		}
+		return "", toObjectErr(err, bucket, object)
+	}
+
+	// Cleanup all the parts if everything else has been safely committed.
+	if err = cleanupUploadedParts(bucket, object, uploadID, fs.storage); err != nil {
+		return "", err
+	}
+
+	// Hold the lock so that two parallel complete-multipart-uploads do not
+	// leave a stale uploads.json behind.
+	nsMutex.Lock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object))
+	defer nsMutex.Unlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object))
+
+	// Validate if there are other incomplete upload-id's present for
+	// the object, if yes do not attempt to delete 'uploads.json'.
+	uploadsJSON, err := readUploadsJSON(bucket, object, fs.storage)
+	if err != nil {
+		return "", toObjectErr(err, minioMetaBucket, object)
+	}
+	// If we have successfully read `uploads.json`, then we proceed to
+	// purge or update `uploads.json`.
+	uploadIDIdx := uploadsJSON.Index(uploadID)
+	if uploadIDIdx != -1 {
+		uploadsJSON.Uploads = append(uploadsJSON.Uploads[:uploadIDIdx], uploadsJSON.Uploads[uploadIDIdx+1:]...)
+	}
+	if len(uploadsJSON.Uploads) > 0 {
+		if err = updateUploadsJSON(bucket, object, uploadsJSON, fs.storage); err != nil {
+			return "", toObjectErr(err, minioMetaBucket, path.Join(mpartMetaPrefix, bucket, object))
+		}
+		// Return success.
+		return s3MD5, nil
+	}
+
+	if err = fs.storage.DeleteFile(minioMetaBucket, path.Join(mpartMetaPrefix, bucket, object, uploadsJSONFile)); err != nil {
+		return "", toObjectErr(err, minioMetaBucket, path.Join(mpartMetaPrefix, bucket, object))
+	}
+
+	// Return md5sum.
+	return s3MD5, nil
+}
+
+// abortMultipartUpload - wrapper for purging an ongoing multipart
+// transaction, deletes uploadID entry from `uploads.json` and purges
+// the directory at '.minio/multipart/bucket/object/uploadID' holding
+// all the upload parts.
+func (fs fsObjects) abortMultipartUpload(bucket, object, uploadID string) error {
+	// Cleanup all uploaded parts.
+	if err := cleanupUploadedParts(bucket, object, uploadID, fs.storage); err != nil {
+		return err
+	}
+
+	// Validate if there are other incomplete upload-id's present for
+	// the object, if yes do not attempt to delete 'uploads.json'.
+	uploadsJSON, err := readUploadsJSON(bucket, object, fs.storage)
+	if err == nil {
+		uploadIDIdx := uploadsJSON.Index(uploadID)
+		if uploadIDIdx != -1 {
+			uploadsJSON.Uploads = append(uploadsJSON.Uploads[:uploadIDIdx], uploadsJSON.Uploads[uploadIDIdx+1:]...)
+		}
+		// There are pending uploads for the same object, preserve
+		// them update 'uploads.json' in-place.
+		if len(uploadsJSON.Uploads) > 0 {
+			err = updateUploadsJSON(bucket, object, uploadsJSON, fs.storage)
+			if err != nil {
+				return toObjectErr(err, bucket, object)
+			}
+			return nil
+		}
+	} // No more pending uploads for the object, we purge the entire
+	// entry at '.minio/multipart/bucket/object'.
+	if err = fs.storage.DeleteFile(minioMetaBucket, path.Join(mpartMetaPrefix, bucket, object, uploadsJSONFile)); err != nil {
+		return toObjectErr(err, minioMetaBucket, path.Join(mpartMetaPrefix, bucket, object))
+	}
+	return nil
+}
+
+// AbortMultipartUpload - aborts an ongoing multipart operation
+// signified by the input uploadID. This is an atomic operation
+// doesn't require clients to initiate multiple such requests.
+//
+// All parts are purged from all disks and reference to the uploadID
+// would be removed from the system, rollback is not possible on this
+// operation.
+//
+// Implements S3 compatible Abort multipart API, slight difference is
+// that this is an atomic idempotent operation. Subsequent calls have
+// no affect and further requests to the same uploadID would not be
+// honored.
+func (fs fsObjects) AbortMultipartUpload(bucket, object, uploadID string) error {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return BucketNameInvalid{Bucket: bucket}
+	}
+	if !fs.isBucketExist(bucket) {
+		return BucketNotFound{Bucket: bucket}
+	}
+	if !IsValidObjectName(object) {
+		return ObjectNameInvalid{Bucket: bucket, Object: object}
+	}
+
+	// Hold lock so that there is no competing complete-multipart-upload or put-object-part.
+	nsMutex.Lock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID))
+	defer nsMutex.Unlock(minioMetaBucket, pathJoin(mpartMetaPrefix, bucket, object, uploadID))
+
+	if !fs.isUploadIDExists(bucket, object, uploadID) {
+		return InvalidUploadID{UploadID: uploadID}
+	}
+
+	err := fs.abortMultipartUpload(bucket, object, uploadID)
+	return err
+}

fs-v1.go

@@ -0,0 +1,435 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"crypto/md5"
+	"encoding/hex"
+	"io"
+	"os"
+	"path"
+	"path/filepath"
+	"sort"
+	"strings"
+	"sync"
+
+	"github.com/minio/minio/pkg/disk"
+	"github.com/minio/minio/pkg/mimedb"
+)
+
+// fsObjects - Implements fs object layer.
+type fsObjects struct {
+	storage            StorageAPI
+	physicalDisk       string
+	listObjectMap      map[listParams][]*treeWalkerFS
+	listObjectMapMutex *sync.Mutex
+}
+
+// newFSObjects - initialize new fs object layer.
+func newFSObjects(disk string) (ObjectLayer, error) {
+	storage, err := newStorageAPI(disk)
+	if err != nil {
+		return nil, err
+	}
+
+	// Runs house keeping code, like creating minioMetaBucket, cleaning up tmp files etc.
+	fsHouseKeeping(storage)
+
+	// Return successfully initialized object layer.
+	return fsObjects{
+		storage:            storage,
+		physicalDisk:       disk,
+		listObjectMap:      make(map[listParams][]*treeWalkerFS),
+		listObjectMapMutex: &sync.Mutex{},
+	}, nil
+}
+
+// StorageInfo - returns underlying storage statistics.
+func (fs fsObjects) StorageInfo() StorageInfo {
+	info, err := disk.GetInfo(fs.physicalDisk)
+	fatalIf(err, "Unable to get disk info "+fs.physicalDisk)
+	return StorageInfo{
+		Total: info.Total,
+		Free:  info.Free,
+	}
+}
+
+/// Bucket operations
+
+// MakeBucket - make a bucket.
+func (fs fsObjects) MakeBucket(bucket string) error {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return BucketNameInvalid{Bucket: bucket}
+	}
+	if err := fs.storage.MakeVol(bucket); err != nil {
+		return toObjectErr(err, bucket)
+	}
+	return nil
+}
+
+// GetBucketInfo - get bucket info.
+func (fs fsObjects) GetBucketInfo(bucket string) (BucketInfo, error) {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return BucketInfo{}, BucketNameInvalid{Bucket: bucket}
+	}
+	vi, err := fs.storage.StatVol(bucket)
+	if err != nil {
+		return BucketInfo{}, toObjectErr(err, bucket)
+	}
+	return BucketInfo{
+		Name:    bucket,
+		Created: vi.Created,
+	}, nil
+}
+
+// ListBuckets - list buckets.
+func (fs fsObjects) ListBuckets() ([]BucketInfo, error) {
+	var bucketInfos []BucketInfo
+	vols, err := fs.storage.ListVols()
+	if err != nil {
+		return nil, toObjectErr(err)
+	}
+	for _, vol := range vols {
+		// StorageAPI can send volume names which are incompatible
+		// with buckets, handle it and skip them.
+		if !IsValidBucketName(vol.Name) {
+			continue
+		}
+		bucketInfos = append(bucketInfos, BucketInfo{
+			Name:    vol.Name,
+			Created: vol.Created,
+		})
+	}
+	sort.Sort(byBucketName(bucketInfos))
+	return bucketInfos, nil
+}
+
+// DeleteBucket - delete a bucket.
+func (fs fsObjects) DeleteBucket(bucket string) error {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return BucketNameInvalid{Bucket: bucket}
+	}
+	if err := fs.storage.DeleteVol(bucket); err != nil {
+		return toObjectErr(err, bucket)
+	}
+	return nil
+}
+
+/// Object Operations
+
+// GetObject - get an object.
+func (fs fsObjects) GetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) error {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return BucketNameInvalid{Bucket: bucket}
+	}
+	// Verify if object is valid.
+	if !IsValidObjectName(object) {
+		return ObjectNameInvalid{Bucket: bucket, Object: object}
+	}
+	var totalLeft = length
+	for totalLeft > 0 {
+		// Figure out the right blockSize as it was encoded before.
+		var curBlockSize int64
+		if blockSizeV1 < totalLeft {
+			curBlockSize = blockSizeV1
+		} else {
+			curBlockSize = totalLeft
+		}
+		buf := make([]byte, curBlockSize)
+		n, err := fs.storage.ReadFile(bucket, object, startOffset, buf)
+		if err != nil {
+			return toObjectErr(err, bucket, object)
+		}
+		_, err = writer.Write(buf[:n])
+		if err != nil {
+			return toObjectErr(err, bucket, object)
+		}
+		totalLeft -= n
+		startOffset += n
+	}
+	return nil
+}
+
+// GetObjectInfo - get object info.
+func (fs fsObjects) GetObjectInfo(bucket, object string) (ObjectInfo, error) {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return ObjectInfo{}, (BucketNameInvalid{Bucket: bucket})
+	}
+	// Verify if object is valid.
+	if !IsValidObjectName(object) {
+		return ObjectInfo{}, (ObjectNameInvalid{Bucket: bucket, Object: object})
+	}
+	fi, err := fs.storage.StatFile(bucket, object)
+	if err != nil {
+		return ObjectInfo{}, toObjectErr(err, bucket, object)
+	}
+	contentType := "application/octet-stream"
+	if objectExt := filepath.Ext(object); objectExt != "" {
+		content, ok := mimedb.DB[strings.ToLower(strings.TrimPrefix(objectExt, "."))]
+		if ok {
+			contentType = content.ContentType
+		}
+	}
+	return ObjectInfo{
+		Bucket:      bucket,
+		Name:        object,
+		ModTime:     fi.ModTime,
+		Size:        fi.Size,
+		IsDir:       fi.Mode.IsDir(),
+		ContentType: contentType,
+		MD5Sum:      "", // Read from metadata.
+	}, nil
+}
+
+// PutObject - create an object.
+func (fs fsObjects) PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (string, error) {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return "", BucketNameInvalid{Bucket: bucket}
+	}
+	if !IsValidObjectName(object) {
+		return "", ObjectNameInvalid{
+			Bucket: bucket,
+			Object: object,
+		}
+	}
+
+	uniqueID := getUUID()
+
+	// Temporary object.
+	tempObj := path.Join(tmpMetaPrefix, uniqueID)
+
+	// Initialize md5 writer.
+	md5Writer := md5.New()
+
+	if size == 0 {
+		// For size 0 we write a 0byte file.
+		_, err := fs.storage.AppendFile(minioMetaBucket, tempObj, []byte(""))
+		if err != nil {
+			return "", toObjectErr(err, bucket, object)
+		}
+	} else {
+		// Allocate buffer.
+		buf := make([]byte, blockSizeV1)
+		for {
+			n, rErr := data.Read(buf)
+			if rErr == io.EOF {
+				break
+			}
+			if rErr != nil {
+				return "", toObjectErr(rErr, bucket, object)
+			}
+			// Update md5 writer.
+			md5Writer.Write(buf[:n])
+			m, wErr := fs.storage.AppendFile(minioMetaBucket, tempObj, buf[:n])
+			if wErr != nil {
+				return "", toObjectErr(wErr, bucket, object)
+			}
+			if m != int64(len(buf[:n])) {
+				return "", toObjectErr(errUnexpected, bucket, object)
+			}
+		}
+	}
+
+	newMD5Hex := hex.EncodeToString(md5Writer.Sum(nil))
+	// md5Hex representation.
+	var md5Hex string
+	if len(metadata) != 0 {
+		md5Hex = metadata["md5Sum"]
+	}
+	if md5Hex != "" {
+		if newMD5Hex != md5Hex {
+			return "", BadDigest{md5Hex, newMD5Hex}
+		}
+	}
+
+	err := fs.storage.RenameFile(minioMetaBucket, tempObj, bucket, object)
+	if err != nil {
+		return "", toObjectErr(err, bucket, object)
+	}
+
+	// Return md5sum, successfully wrote object.
+	return newMD5Hex, nil
+}
+
+func (fs fsObjects) DeleteObject(bucket, object string) error {
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return BucketNameInvalid{Bucket: bucket}
+	}
+	if !IsValidObjectName(object) {
+		return ObjectNameInvalid{Bucket: bucket, Object: object}
+	}
+	if err := fs.storage.DeleteFile(bucket, object); err != nil {
+		return toObjectErr(err, bucket, object)
+	}
+	return nil
+}
+
+// Checks whether bucket exists.
+func isBucketExist(storage StorageAPI, bucketName string) bool {
+	// Check whether bucket exists.
+	_, err := storage.StatVol(bucketName)
+	if err != nil {
+		if err == errVolumeNotFound {
+			return false
+		}
+		errorIf(err, "Stat failed on bucket "+bucketName+".")
+		return false
+	}
+	return true
+}
+
+func (fs fsObjects) listObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, error) {
+	// Convert entry to FileInfo
+	entryToFileInfo := func(entry string) (fileInfo FileInfo, err error) {
+		if strings.HasSuffix(entry, slashSeparator) {
+			// Object name needs to be full path.
+			fileInfo.Name = entry
+			fileInfo.Mode = os.ModeDir
+			return
+		}
+		if fileInfo, err = fs.storage.StatFile(bucket, entry); err != nil {
+			return
+		}
+		// Object name needs to be full path.
+		fileInfo.Name = entry
+		return
+	}
+
+	// Verify if bucket is valid.
+	if !IsValidBucketName(bucket) {
+		return ListObjectsInfo{}, BucketNameInvalid{Bucket: bucket}
+	}
+	// Verify if bucket exists.
+	if !isBucketExist(fs.storage, bucket) {
+		return ListObjectsInfo{}, BucketNotFound{Bucket: bucket}
+	}
+	if !IsValidObjectPrefix(prefix) {
+		return ListObjectsInfo{}, ObjectNameInvalid{Bucket: bucket, Object: prefix}
+	}
+	// Verify if delimiter is anything other than '/', which we do not support.
+	if delimiter != "" && delimiter != slashSeparator {
+		return ListObjectsInfo{}, UnsupportedDelimiter{
+			Delimiter: delimiter,
+		}
+	}
+	// Verify if marker has prefix.
+	if marker != "" {
+		if !strings.HasPrefix(marker, prefix) {
+			return ListObjectsInfo{}, InvalidMarkerPrefixCombination{
+				Marker: marker,
+				Prefix: prefix,
+			}
+		}
+	}
+
+	// With max keys of zero we have reached eof, return right here.
+	if maxKeys == 0 {
+		return ListObjectsInfo{}, nil
+	}
+
+	// For delimiter and prefix as '/' we do not list anything at all
+	// since according to s3 spec we stop at the 'delimiter'
+	// along // with the prefix. On a flat namespace with 'prefix'
+	// as '/' we don't have any entries, since all the keys are
+	// of form 'keyName/...'
+	if delimiter == slashSeparator && prefix == slashSeparator {
+		return ListObjectsInfo{}, nil
+	}
+
+	// Over flowing count - reset to maxObjectList.
+	if maxKeys < 0 || maxKeys > maxObjectList {
+		maxKeys = maxObjectList
+	}
+
+	// Default is recursive, if delimiter is set then list non recursive.
+	recursive := true
+	if delimiter == slashSeparator {
+		recursive = false
+	}
+
+	walker := fs.lookupTreeWalk(listParams{bucket, recursive, marker, prefix})
+	if walker == nil {
+		walker = fs.startTreeWalk(bucket, prefix, marker, recursive, func(bucket, object string) bool {
+			return !strings.HasSuffix(object, slashSeparator)
+		})
+	}
+	var fileInfos []FileInfo
+	var eof bool
+	var nextMarker string
+	for i := 0; i < maxKeys; {
+		walkResult, ok := <-walker.ch
+		if !ok {
+			// Closed channel.
+			eof = true
+			break
+		}
+		// For any walk error return right away.
+		if walkResult.err != nil {
+			// File not found is a valid case.
+			if walkResult.err == errFileNotFound {
+				return ListObjectsInfo{}, nil
+			}
+			return ListObjectsInfo{}, toObjectErr(walkResult.err, bucket, prefix)
+		}
+		fileInfo, err := entryToFileInfo(walkResult.entry)
+		if err != nil {
+			return ListObjectsInfo{}, nil
+		}
+		nextMarker = fileInfo.Name
+		fileInfos = append(fileInfos, fileInfo)
+		if walkResult.end {
+			eof = true
+			break
+		}
+		i++
+	}
+	params := listParams{bucket, recursive, nextMarker, prefix}
+	if !eof {
+		fs.saveTreeWalk(params, walker)
+	}
+
+	result := ListObjectsInfo{IsTruncated: !eof}
+	for _, fileInfo := range fileInfos {
+		// With delimiter set we fill in NextMarker and Prefixes.
+		if delimiter == slashSeparator {
+			result.NextMarker = fileInfo.Name
+			if fileInfo.Mode.IsDir() {
+				result.Prefixes = append(result.Prefixes, fileInfo.Name)
+				continue
+			}
+		}
+		result.Objects = append(result.Objects, ObjectInfo{
+			Name:    fileInfo.Name,
+			ModTime: fileInfo.ModTime,
+			Size:    fileInfo.Size,
+			IsDir:   false,
+		})
+	}
+	return result, nil
+}
+
+// ListObjects - list all objects.
+func (fs fsObjects) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, error) {
+	return fs.listObjects(bucket, prefix, marker, delimiter, maxKeys)
+}

fs.go

@@ -1,112 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"os"
-	"path/filepath"
-	"sync"
-
-	"github.com/minio/minio/pkg/disk"
-	"github.com/minio/minio/pkg/probe"
-)
-
-// listObjectParams - list object params used for list object map
-type listObjectParams struct {
-	bucket    string
-	delimiter string
-	marker    string
-	prefix    string
-}
-
-// listMultipartObjectParams - list multipart object params used for list multipart object map
-type listMultipartObjectParams struct {
-	bucket         string
-	delimiter      string
-	keyMarker      string
-	prefix         string
-	uploadIDMarker string
-}
-
-// Filesystem - local variables
-type Filesystem struct {
-	diskPath                    string
-	minFreeDisk                 int64
-	rwLock                      *sync.RWMutex
-	listObjectMap               map[listObjectParams][]*treeWalker
-	listObjectMapMutex          *sync.Mutex
-	listMultipartObjectMap      map[listMultipartObjectParams][]<-chan multipartObjectInfo
-	listMultipartObjectMapMutex *sync.Mutex
-}
-
-// newFS instantiate a new filesystem.
-func newFS(diskPath string) (ObjectAPI, *probe.Error) {
-	fs := &Filesystem{
-		rwLock: &sync.RWMutex{},
-	}
-	fs.diskPath = diskPath
-
-	/// Defaults
-	// Minium free disk required for i/o operations to succeed.
-	fs.minFreeDisk = 5
-
-	// Initialize list object map.
-	fs.listObjectMap = make(map[listObjectParams][]*treeWalker)
-	fs.listObjectMapMutex = &sync.Mutex{}
-
-	// Initialize list multipart map.
-	fs.listMultipartObjectMap = make(map[listMultipartObjectParams][]<-chan multipartObjectInfo)
-	fs.listMultipartObjectMapMutex = &sync.Mutex{}
-
-	// Return here.
-	return fs, nil
-}
-
-func (fs Filesystem) checkBucketArg(bucket string) (string, error) {
-	if !IsValidBucketName(bucket) {
-		return "", BucketNameInvalid{Bucket: bucket}
-	}
-	bucket = getActualBucketname(fs.diskPath, bucket)
-	if status, e := isDirExist(filepath.Join(fs.diskPath, bucket)); !status {
-		if e == nil {
-			return "", BucketNotFound{Bucket: bucket}
-		} else if os.IsNotExist(e) {
-			return "", BucketNotFound{Bucket: bucket}
-		} else {
-			return "", e
-		}
-	}
-	return bucket, nil
-}
-
-func checkDiskFree(diskPath string, minFreeDisk int64) error {
-	di, e := disk.GetInfo(diskPath)
-	if e != nil {
-		return e
-	}
-	// Remove 5% from total space for cumulative disk space used for journalling, inodes etc.
-	availableDiskSpace := (float64(di.Free) / (float64(di.Total) - (0.05 * float64(di.Total)))) * 100
-	if int64(availableDiskSpace) <= minFreeDisk {
-		return RootPathFull{Path: diskPath}
-	}
-	return nil
-}
-
-// GetRootPath - get root path.
-func (fs Filesystem) GetRootPath() string {
-	return fs.diskPath
-}

fs_api_suite_test.go

@@ -1,423 +0,0 @@
-/*
- * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
-	"bytes"
-	"crypto/md5"
-	"encoding/hex"
-	"io"
-	"math/rand"
-	"strconv"
-
-	"gopkg.in/check.v1"
-)
-
-// APITestSuite - collection of API tests
-func APITestSuite(c *check.C, create func() ObjectAPI) {
-	testMakeBucket(c, create)
-	testMultipleObjectCreation(c, create)
-	testPaging(c, create)
-	testObjectOverwriteWorks(c, create)
-	testNonExistantBucketOperations(c, create)
-	testBucketRecreateFails(c, create)
-	testPutObjectInSubdir(c, create)
-	testListBuckets(c, create)
-	testListBucketsOrder(c, create)
-	testListObjectsTestsForNonExistantBucket(c, create)
-	testNonExistantObjectInBucket(c, create)
-	testGetDirectoryReturnsObjectNotFound(c, create)
-	testDefaultContentType(c, create)
-	testMultipartObjectCreation(c, create)
-	testMultipartObjectAbort(c, create)
-}
-
-func testMakeBucket(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-}
-
-func testMultipartObjectCreation(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-	uploadID, err := fs.NewMultipartUpload("bucket", "key")
-	c.Assert(err, check.IsNil)
-
-	completedParts := completeMultipartUpload{}
-	for i := 1; i <= 10; i++ {
-		randomPerm := rand.Perm(10)
-		randomString := ""
-		for _, num := range randomPerm {
-			randomString = randomString + strconv.Itoa(num)
-		}
-
-		hasher := md5.New()
-		hasher.Write([]byte(randomString))
-		expectedMD5Sumhex := hex.EncodeToString(hasher.Sum(nil))
-
-		var calculatedMD5sum string
-		calculatedMD5sum, err = fs.PutObjectPart("bucket", "key", uploadID, i, int64(len(randomString)), bytes.NewBufferString(randomString), expectedMD5Sumhex)
-		c.Assert(err, check.IsNil)
-		c.Assert(calculatedMD5sum, check.Equals, expectedMD5Sumhex)
-		completedParts.Parts = append(completedParts.Parts, completePart{PartNumber: i, ETag: calculatedMD5sum})
-	}
-	objInfo, err := fs.CompleteMultipartUpload("bucket", "key", uploadID, completedParts.Parts)
-	c.Assert(err, check.IsNil)
-	c.Assert(objInfo.MD5Sum, check.Equals, "3605d84b1c43b1a664aa7c0d5082d271-10")
-}
-
-func testMultipartObjectAbort(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-	uploadID, err := fs.NewMultipartUpload("bucket", "key")
-	c.Assert(err, check.IsNil)
-
-	parts := make(map[int]string)
-	metadata := make(map[string]string)
-	for i := 1; i <= 10; i++ {
-		randomPerm := rand.Perm(10)
-		randomString := ""
-		for _, num := range randomPerm {
-			randomString = randomString + strconv.Itoa(num)
-		}
-
-		hasher := md5.New()
-		hasher.Write([]byte(randomString))
-		expectedMD5Sumhex := hex.EncodeToString(hasher.Sum(nil))
-
-		metadata["md5"] = expectedMD5Sumhex
-		var calculatedMD5sum string
-		calculatedMD5sum, err = fs.PutObjectPart("bucket", "key", uploadID, i, int64(len(randomString)), bytes.NewBufferString(randomString), expectedMD5Sumhex)
-		c.Assert(err, check.IsNil)
-		c.Assert(calculatedMD5sum, check.Equals, expectedMD5Sumhex)
-		parts[i] = expectedMD5Sumhex
-	}
-	err = fs.AbortMultipartUpload("bucket", "key", uploadID)
-	c.Assert(err, check.IsNil)
-}
-
-func testMultipleObjectCreation(c *check.C, create func() ObjectAPI) {
-	objects := make(map[string][]byte)
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-	for i := 0; i < 10; i++ {
-		randomPerm := rand.Perm(10)
-		randomString := ""
-		for _, num := range randomPerm {
-			randomString = randomString + strconv.Itoa(num)
-		}
-
-		hasher := md5.New()
-		hasher.Write([]byte(randomString))
-		expectedMD5Sumhex := hex.EncodeToString(hasher.Sum(nil))
-
-		key := "obj" + strconv.Itoa(i)
-		objects[key] = []byte(randomString)
-		metadata := make(map[string]string)
-		metadata["md5Sum"] = expectedMD5Sumhex
-		objInfo, err := fs.PutObject("bucket", key, int64(len(randomString)), bytes.NewBufferString(randomString), metadata)
-		c.Assert(err, check.IsNil)
-		c.Assert(objInfo.MD5Sum, check.Equals, expectedMD5Sumhex)
-	}
-
-	for key, value := range objects {
-		var byteBuffer bytes.Buffer
-		r, err := fs.GetObject("bucket", key, 0)
-		c.Assert(err, check.IsNil)
-		_, e := io.Copy(&byteBuffer, r)
-		c.Assert(e, check.IsNil)
-		c.Assert(byteBuffer.Bytes(), check.DeepEquals, value)
-		c.Assert(r.Close(), check.IsNil)
-
-		objInfo, err := fs.GetObjectInfo("bucket", key)
-		c.Assert(err, check.IsNil)
-		c.Assert(objInfo.Size, check.Equals, int64(len(value)))
-		r.Close()
-	}
-}
-
-func testPaging(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	fs.MakeBucket("bucket")
-	result, err := fs.ListObjects("bucket", "", "", "", 0)
-	c.Assert(err, check.IsNil)
-	c.Assert(len(result.Objects), check.Equals, 0)
-	c.Assert(result.IsTruncated, check.Equals, false)
-	// check before paging occurs
-	for i := 0; i < 5; i++ {
-		key := "obj" + strconv.Itoa(i)
-		_, err = fs.PutObject("bucket", key, int64(len(key)), bytes.NewBufferString(key), nil)
-		c.Assert(err, check.IsNil)
-		result, err = fs.ListObjects("bucket", "", "", "", 5)
-		c.Assert(err, check.IsNil)
-		c.Assert(len(result.Objects), check.Equals, i+1)
-		c.Assert(result.IsTruncated, check.Equals, false)
-	}
-	// check after paging occurs pages work
-	for i := 6; i <= 10; i++ {
-		key := "obj" + strconv.Itoa(i)
-		_, err = fs.PutObject("bucket", key, int64(len(key)), bytes.NewBufferString(key), nil)
-		c.Assert(err, check.IsNil)
-		result, err = fs.ListObjects("bucket", "obj", "", "", 5)
-		c.Assert(err, check.IsNil)
-		c.Assert(len(result.Objects), check.Equals, 5)
-		c.Assert(result.IsTruncated, check.Equals, true)
-	}
-	// check paging with prefix at end returns less objects
-	{
-		_, err = fs.PutObject("bucket", "newPrefix", int64(len("prefix1")), bytes.NewBufferString("prefix1"), nil)
-		c.Assert(err, check.IsNil)
-		_, err = fs.PutObject("bucket", "newPrefix2", int64(len("prefix2")), bytes.NewBufferString("prefix2"), nil)
-		c.Assert(err, check.IsNil)
-		result, err = fs.ListObjects("bucket", "new", "", "", 5)
-		c.Assert(err, check.IsNil)
-		c.Assert(len(result.Objects), check.Equals, 2)
-	}
-
-	// check ordering of pages
-	{
-		result, err = fs.ListObjects("bucket", "", "", "", 1000)
-		c.Assert(err, check.IsNil)
-		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix")
-		c.Assert(result.Objects[1].Name, check.Equals, "newPrefix2")
-		c.Assert(result.Objects[2].Name, check.Equals, "obj0")
-		c.Assert(result.Objects[3].Name, check.Equals, "obj1")
-		c.Assert(result.Objects[4].Name, check.Equals, "obj10")
-	}
-
-	// check delimited results with delimiter and prefix
-	{
-		_, err = fs.PutObject("bucket", "this/is/delimited", int64(len("prefix1")), bytes.NewBufferString("prefix1"), nil)
-		c.Assert(err, check.IsNil)
-		_, err = fs.PutObject("bucket", "this/is/also/a/delimited/file", int64(len("prefix2")), bytes.NewBufferString("prefix2"), nil)
-		c.Assert(err, check.IsNil)
-		result, err = fs.ListObjects("bucket", "this/is/", "", "/", 10)
-		c.Assert(err, check.IsNil)
-		c.Assert(len(result.Objects), check.Equals, 1)
-		c.Assert(result.Prefixes[0], check.Equals, "this/is/also/")
-	}
-
-	// check delimited results with delimiter without prefix
-	{
-		result, err = fs.ListObjects("bucket", "", "", "/", 1000)
-		c.Assert(err, check.IsNil)
-		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix")
-		c.Assert(result.Objects[1].Name, check.Equals, "newPrefix2")
-		c.Assert(result.Objects[2].Name, check.Equals, "obj0")
-		c.Assert(result.Objects[3].Name, check.Equals, "obj1")
-		c.Assert(result.Objects[4].Name, check.Equals, "obj10")
-		c.Assert(result.Prefixes[0], check.Equals, "this/")
-	}
-
-	// check results with Marker
-	{
-		result, err = fs.ListObjects("bucket", "", "newPrefix", "", 3)
-		c.Assert(err, check.IsNil)
-		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix2")
-		c.Assert(result.Objects[1].Name, check.Equals, "obj0")
-		c.Assert(result.Objects[2].Name, check.Equals, "obj1")
-	}
-	// check ordering of results with prefix
-	{
-		result, err = fs.ListObjects("bucket", "obj", "", "", 1000)
-		c.Assert(err, check.IsNil)
-		c.Assert(result.Objects[0].Name, check.Equals, "obj0")
-		c.Assert(result.Objects[1].Name, check.Equals, "obj1")
-		c.Assert(result.Objects[2].Name, check.Equals, "obj10")
-		c.Assert(result.Objects[3].Name, check.Equals, "obj2")
-		c.Assert(result.Objects[4].Name, check.Equals, "obj3")
-	}
-	// check ordering of results with prefix and no paging
-	{
-		result, err = fs.ListObjects("bucket", "new", "", "", 5)
-		c.Assert(err, check.IsNil)
-		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix")
-		c.Assert(result.Objects[1].Name, check.Equals, "newPrefix2")
-	}
-}
-
-func testObjectOverwriteWorks(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-
-	_, err = fs.PutObject("bucket", "object", int64(len("one")), bytes.NewBufferString("one"), nil)
-	c.Assert(err, check.IsNil)
-	// c.Assert(md5Sum1hex, check.Equals, objInfo.MD5Sum)
-
-	_, err = fs.PutObject("bucket", "object", int64(len("three")), bytes.NewBufferString("three"), nil)
-	c.Assert(err, check.IsNil)
-
-	var bytesBuffer bytes.Buffer
-	r, err := fs.GetObject("bucket", "object", 0)
-	c.Assert(err, check.IsNil)
-	_, e := io.Copy(&bytesBuffer, r)
-	c.Assert(e, check.IsNil)
-	c.Assert(string(bytesBuffer.Bytes()), check.Equals, "three")
-	c.Assert(r.Close(), check.IsNil)
-}
-
-func testNonExistantBucketOperations(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	_, err := fs.PutObject("bucket", "object", int64(len("one")), bytes.NewBufferString("one"), nil)
-	c.Assert(err, check.Not(check.IsNil))
-}
-
-func testBucketRecreateFails(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("string")
-	c.Assert(err, check.IsNil)
-	err = fs.MakeBucket("string")
-	c.Assert(err, check.Not(check.IsNil))
-}
-
-func testPutObjectInSubdir(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-
-	_, err = fs.PutObject("bucket", "dir1/dir2/object", int64(len("hello world")), bytes.NewBufferString("hello world"), nil)
-	c.Assert(err, check.IsNil)
-
-	var bytesBuffer bytes.Buffer
-	r, err := fs.GetObject("bucket", "dir1/dir2/object", 0)
-	c.Assert(err, check.IsNil)
-	n, e := io.Copy(&bytesBuffer, r)
-	c.Assert(e, check.IsNil)
-	c.Assert(len(bytesBuffer.Bytes()), check.Equals, len("hello world"))
-	c.Assert(int64(len(bytesBuffer.Bytes())), check.Equals, int64(n))
-	c.Assert(r.Close(), check.IsNil)
-}
-
-func testListBuckets(c *check.C, create func() ObjectAPI) {
-	fs := create()
-
-	// test empty list
-	buckets, err := fs.ListBuckets()
-	c.Assert(err, check.IsNil)
-	c.Assert(len(buckets), check.Equals, 0)
-
-	// add one and test exists
-	err = fs.MakeBucket("bucket1")
-	c.Assert(err, check.IsNil)
-
-	buckets, err = fs.ListBuckets()
-	c.Assert(len(buckets), check.Equals, 1)
-	c.Assert(err, check.IsNil)
-
-	// add two and test exists
-	err = fs.MakeBucket("bucket2")
-	c.Assert(err, check.IsNil)
-
-	buckets, err = fs.ListBuckets()
-	c.Assert(len(buckets), check.Equals, 2)
-	c.Assert(err, check.IsNil)
-
-	// add three and test exists + prefix
-	err = fs.MakeBucket("bucket22")
-
-	buckets, err = fs.ListBuckets()
-	c.Assert(len(buckets), check.Equals, 3)
-	c.Assert(err, check.IsNil)
-}
-
-func testListBucketsOrder(c *check.C, create func() ObjectAPI) {
-	// if implementation contains a map, order of map keys will vary.
-	// this ensures they return in the same order each time
-	for i := 0; i < 10; i++ {
-		fs := create()
-		// add one and test exists
-		err := fs.MakeBucket("bucket1")
-		c.Assert(err, check.IsNil)
-		err = fs.MakeBucket("bucket2")
-		c.Assert(err, check.IsNil)
-		buckets, err := fs.ListBuckets()
-		c.Assert(err, check.IsNil)
-		c.Assert(len(buckets), check.Equals, 2)
-		c.Assert(buckets[0].Name, check.Equals, "bucket1")
-		c.Assert(buckets[1].Name, check.Equals, "bucket2")
-	}
-}
-
-func testListObjectsTestsForNonExistantBucket(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	result, err := fs.ListObjects("bucket", "", "", "", 1000)
-	c.Assert(err, check.Not(check.IsNil))
-	c.Assert(result.IsTruncated, check.Equals, false)
-	c.Assert(len(result.Objects), check.Equals, 0)
-}
-
-func testNonExistantObjectInBucket(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-
-	_, err = fs.GetObject("bucket", "dir1", 0)
-	c.Assert(err, check.Not(check.IsNil))
-	switch err := err.ToGoError().(type) {
-	case ObjectNotFound:
-		c.Assert(err, check.ErrorMatches, "Object not found: bucket#dir1")
-	default:
-		c.Assert(err, check.Equals, "fails")
-	}
-}
-
-func testGetDirectoryReturnsObjectNotFound(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-
-	_, err = fs.PutObject("bucket", "dir1/dir2/object", int64(len("hello world")), bytes.NewBufferString("hello world"), nil)
-	c.Assert(err, check.IsNil)
-
-	_, err = fs.GetObject("bucket", "dir1", 0)
-	switch err := err.ToGoError().(type) {
-	case ObjectExistsAsPrefix:
-		c.Assert(err.Bucket, check.Equals, "bucket")
-		c.Assert(err.Prefix, check.Equals, "dir1")
-	default:
-		// force a failure with a line number
-		c.Assert(err.Error(), check.Equals, "Object exists on : bucket as prefix dir1")
-	}
-
-	_, err = fs.GetObject("bucket", "dir1/", 0)
-	switch err := err.ToGoError().(type) {
-	case ObjectExistsAsPrefix:
-		c.Assert(err.Bucket, check.Equals, "bucket")
-		c.Assert(err.Prefix, check.Equals, "dir1/")
-	default:
-		// force a failure with a line number
-		c.Assert(err.Error(), check.Equals, "Object exists on : bucket as prefix dir1")
-	}
-}
-
-func testDefaultContentType(c *check.C, create func() ObjectAPI) {
-	fs := create()
-	err := fs.MakeBucket("bucket")
-	c.Assert(err, check.IsNil)
-
-	// Test empty
-	_, err = fs.PutObject("bucket", "one", int64(len("one")), bytes.NewBufferString("one"), nil)
-	c.Assert(err, check.IsNil)
-	objInfo, err := fs.GetObjectInfo("bucket", "one")
-	c.Assert(err, check.IsNil)
-	c.Assert(objInfo.ContentType, check.Equals, "application/octet-stream")
-}

generic-handlers.go

@@ -58,10 +58,10 @@ func (h redirectHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	// Re-direction handled specifically for browsers.
 	if strings.Contains(r.Header.Get("User-Agent"), "Mozilla") {
 		// '/' is redirected to 'locationPrefix/'
-		// '/rpc' is redirected to 'locationPrefix/rpc'
+		// '/webrpc' is redirected to 'locationPrefix/webrpc'
 		// '/login' is redirected to 'locationPrefix/login'
 		switch r.URL.Path {
-		case "/", "/rpc", "/login", "/favicon.ico":
+		case "/", "/webrpc", "/login", "/favicon.ico":
 			location := h.locationPrefix + r.URL.Path
 			// Redirect to new location.
 			http.Redirect(w, r, location, http.StatusTemporaryRedirect)
@@ -198,6 +198,7 @@ func setCorsHandler(h http.Handler) http.Handler {
 		AllowedOrigins: []string{"*"},
 		AllowedMethods: []string{"GET", "HEAD", "POST", "PUT"},
 		AllowedHeaders: []string{"*"},
+		ExposedHeaders: []string{"ETag"},
 	})
 	return c.Handler(h)
 }

globals.go

@@ -16,11 +16,7 @@
 
 package main
 
-import (
-	"github.com/fatih/color"
-	"github.com/minio/cli"
-	"github.com/minio/mc/pkg/console"
-)
+import "github.com/fatih/color"
 
 // Global constants for Minio.
 const (
@@ -35,11 +31,13 @@ const (
 	globalMinioCertFile      = "public.crt"
 	globalMinioKeyFile       = "private.key"
 	globalMinioConfigFile    = "config.json"
+	globalMinioProfilePath   = "profile"
+	// Add new global values here.
 )
 
 var (
 	globalQuiet = false // Quiet flag set via command line
-	globalDebug = false // Debug flag set via command line
+	globalTrace = false // Trace flag set via environment setting.
 	// Add new global flags here.
 )
 
@@ -49,22 +47,3 @@ var (
 	colorWhite   = color.New(color.FgWhite, color.Bold).SprintfFunc()
 	colorGreen   = color.New(color.FgGreen, color.Bold).SprintfFunc()
 )
-
-// Set global states. NOTE: It is deliberately kept monolithic to
-// ensure we dont miss out any flags.
-func setGlobals(quiet, debug bool) {
-	globalQuiet = quiet
-	globalDebug = debug
-	// Enable debug messages if requested.
-	if globalDebug {
-		console.DebugPrint = true
-	}
-}
-
-// Set global states. NOTE: It is deliberately kept monolithic to
-// ensure we dont miss out any flags.
-func setGlobalsFromContext(ctx *cli.Context) {
-	quiet := ctx.Bool("quiet") || ctx.GlobalBool("quiet")
-	debug := ctx.Bool("debug") || ctx.GlobalBool("debug")
-	setGlobals(quiet, debug)
-}

handler-utils.go

@@ -0,0 +1,54 @@
+/*
+ * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"io"
+)
+
+// validates location constraint from the request body.
+// the location value in the request body should match the Region in serverConfig.
+// other values of location are not accepted.
+// make bucket fails in such cases.
+func isValidLocationContraint(reqBody io.Reader, serverRegion string) APIErrorCode {
+	var locationContraint createBucketLocationConfiguration
+	var errCode APIErrorCode
+	errCode = ErrNone
+	e := xmlDecoder(reqBody, &locationContraint)
+	if e != nil {
+		if e == io.EOF {
+			// Do nothing.
+			// failed due to empty body. The location will be set to default value from the serverConfig.
+			// this is valid.
+			errCode = ErrNone
+		} else {
+			// Failed due to malformed configuration.
+			errCode = ErrMalformedXML
+			//writeErrorResponse(w, r, ErrMalformedXML, r.URL.Path)
+		}
+	} else {
+		// Region obtained from the body.
+		// It should be equal to Region in serverConfig.
+		// Else ErrInvalidRegion returned.
+		// For empty value location will be to set to  default value from the serverConfig.
+		if locationContraint.Location != "" && serverRegion != locationContraint.Location {
+			//writeErrorResponse(w, r, ErrInvalidRegion, r.URL.Path)
+			errCode = ErrInvalidRegion
+		}
+	}
+	return errCode
+}

handler-utils_test.go

@@ -0,0 +1,66 @@
+/*
+ * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"bytes"
+	"encoding/xml"
+	"io/ioutil"
+	"net/http"
+	"testing"
+)
+
+// Tests validate bucket LocationConstraint.
+func TestIsValidLocationContraint(t *testing.T) {
+	// generates the input request with XML bucket configuration set to the request body.
+	createExpectedRequest := func(req *http.Request, location string) (*http.Request, error) {
+		createBucketConfig := createBucketLocationConfiguration{}
+		createBucketConfig.Location = location
+		var createBucketConfigBytes []byte
+		createBucketConfigBytes, e := xml.Marshal(createBucketConfig)
+		if e != nil {
+			return nil, e
+		}
+		createBucketConfigBuffer := bytes.NewBuffer(createBucketConfigBytes)
+		req.Body = ioutil.NopCloser(createBucketConfigBuffer)
+		return req, nil
+	}
+
+	testCases := []struct {
+		locationForInputRequest string
+		serverConfigRegion      string
+		expectedCode            APIErrorCode
+	}{
+		// Test case - 1.
+		{"us-east-1", "us-east-1", ErrNone},
+		// Test case - 2.
+		// In case of empty request body ErrNone is returned.
+		{"", "us-east-1", ErrNone},
+		// Test case - 3.
+		{"eu-central-1", "us-east-1", ErrInvalidRegion},
+	}
+	for i, testCase := range testCases {
+		inputRequest, e := createExpectedRequest(&http.Request{}, testCase.locationForInputRequest)
+		if e != nil {
+			t.Fatalf("Test %d: Failed to Marshal bucket configuration", i+1)
+		}
+		actualCode := isValidLocationContraint(inputRequest.Body, testCase.serverConfigRegion)
+		if testCase.expectedCode != actualCode {
+			t.Errorf("Test %d: Expected the APIErrCode to be %d, but instead found %d", i+1, testCase.expectedCode, actualCode)
+		}
+	}
+}

httprange.go

@@ -21,14 +21,19 @@ import (
 	"fmt"
 	"strconv"
 	"strings"
-
-	"github.com/minio/minio/pkg/probe"
 )
 
 const (
 	b = "bytes="
 )
 
+// InvalidRange - invalid range
+type InvalidRange struct{}
+
+func (e InvalidRange) Error() string {
+	return "The requested range is not satisfiable"
+}
+
 // HttpRange specifies the byte range to be sent to the client.
 type httpRange struct {
 	start, length, size int64
@@ -40,7 +45,7 @@ func (r *httpRange) String() string {
 }
 
 // Grab new range from request header
-func getRequestedRange(hrange string, size int64) (*httpRange, *probe.Error) {
+func getRequestedRange(hrange string, size int64) (*httpRange, error) {
 	r := &httpRange{
 		start:  0,
 		length: 0,
@@ -50,16 +55,16 @@ func getRequestedRange(hrange string, size int64) (*httpRange, *probe.Error) {
 	if hrange != "" {
 		err := r.parseRange(hrange)
 		if err != nil {
-			return nil, err.Trace()
+			return nil, err
 		}
 	}
 	return r, nil
 }
 
-func (r *httpRange) parse(ra string) *probe.Error {
+func (r *httpRange) parse(ra string) error {
 	i := strings.Index(ra, "-")
 	if i < 0 {
-		return probe.NewError(InvalidRange{})
+		return InvalidRange{}
 	}
 	start, end := strings.TrimSpace(ra[:i]), strings.TrimSpace(ra[i+1:])
 	if start == "" {
@@ -67,7 +72,7 @@ func (r *httpRange) parse(ra string) *probe.Error {
 		// range start relative to the end of the file.
 		i, err := strconv.ParseInt(end, 10, 64)
 		if err != nil {
-			return probe.NewError(InvalidRange{})
+			return InvalidRange{}
 		}
 		if i > r.size {
 			i = r.size
@@ -77,7 +82,7 @@ func (r *httpRange) parse(ra string) *probe.Error {
 	} else {
 		i, err := strconv.ParseInt(start, 10, 64)
 		if err != nil || i > r.size || i < 0 {
-			return probe.NewError(InvalidRange{})
+			return InvalidRange{}
 		}
 		r.start = i
 		if end == "" {
@@ -86,7 +91,7 @@ func (r *httpRange) parse(ra string) *probe.Error {
 		} else {
 			i, err := strconv.ParseInt(end, 10, 64)
 			if err != nil || r.start > i {
-				return probe.NewError(InvalidRange{})
+				return InvalidRange{}
 			}
 			if i >= r.size {
 				i = r.size - 1
@@ -98,26 +103,26 @@ func (r *httpRange) parse(ra string) *probe.Error {
 }
 
 // parseRange parses a Range header string as per RFC 2616.
-func (r *httpRange) parseRange(s string) *probe.Error {
+func (r *httpRange) parseRange(s string) error {
 	if s == "" {
-		return probe.NewError(errors.New("header not present"))
+		return errors.New("header not present")
 	}
 	if !strings.HasPrefix(s, b) {
-		return probe.NewError(InvalidRange{})
+		return InvalidRange{}
 	}
 
 	ras := strings.Split(s[len(b):], ",")
 	if len(ras) == 0 {
-		return probe.NewError(errors.New("invalid request"))
+		return errors.New("invalid request")
 	}
 	// Just pick the first one and ignore the rest, we only support one range per object
 	if len(ras) > 1 {
-		return probe.NewError(errors.New("multiple ranges specified"))
+		return errors.New("multiple ranges specified")
 	}
 
 	ra := strings.TrimSpace(ras[0])
 	if ra == "" {
-		return probe.NewError(InvalidRange{})
+		return InvalidRange{}
 	}
 	return r.parse(ra)
 }

logger-console-hook.go

@@ -20,7 +20,6 @@ import (
 	"io/ioutil"
 
 	"github.com/Sirupsen/logrus"
-	"github.com/minio/minio/pkg/probe"
 )
 
 // consoleLogger - default logger if not other logging is enabled.
@@ -37,10 +36,11 @@ func enableConsoleLogger() {
 		log.Out = ioutil.Discard
 		return
 	}
+
 	// log.Out and log.Formatter use the default versions.
 	// Only set specific log level.
-	lvl, e := logrus.ParseLevel(clogger.Level)
-	fatalIf(probe.NewError(e), "Unknown log level detected, please fix your console logger configuration.", nil)
+	lvl, err := logrus.ParseLevel(clogger.Level)
+	fatalIf(err, "Unknown log level found in the config file.")
 
 	log.Level = lvl
 }

logger-file-hook.go

@@ -21,7 +21,6 @@ import (
 	"os"
 
 	"github.com/Sirupsen/logrus"
-	"github.com/minio/minio/pkg/probe"
 )
 
 type fileLogger struct {
@@ -34,15 +33,24 @@ type localFile struct {
 	*os.File
 }
 
-func enableFileLogger(filename string) {
-	file, e := os.OpenFile(filename, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
-	fatalIf(probe.NewError(e), "Unable to open log file.", nil)
+func enableFileLogger() {
+	flogger := serverConfig.GetFileLogger()
+	if !flogger.Enable || flogger.Filename == "" {
+		return
+	}
+
+	file, err := os.OpenFile(flogger.Filename, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
+	fatalIf(err, "Unable to open log file.")
 
 	// Add a local file hook.
 	log.Hooks.Add(&localFile{file})
+
+	lvl, err := logrus.ParseLevel(flogger.Level)
+	fatalIf(err, "Unknown log level found in the config file.")
+
 	// Set default JSON formatter.
 	log.Formatter = new(logrus.JSONFormatter)
-	log.Level = logrus.InfoLevel // Minimum log level.
+	log.Level = lvl // Minimum log level.
 }
 
 // Fire fires the file logger hook and logs to the file.
@@ -56,14 +64,11 @@ func (l *localFile) Fire(entry *logrus.Entry) error {
 	return nil
 }
 
-// Levels -
+// Levels - indicate log levels supported.
 func (l *localFile) Levels() []logrus.Level {
 	return []logrus.Level{
 		logrus.PanicLevel,
 		logrus.FatalLevel,
 		logrus.ErrorLevel,
-		logrus.WarnLevel,
-		logrus.InfoLevel,
-		logrus.DebugLevel,
 	}
 }

logger-syslog-hook_nix.go

@@ -23,7 +23,6 @@ import (
 	"log/syslog"
 
 	"github.com/Sirupsen/logrus"
-	"github.com/minio/minio/pkg/probe"
 )
 
 type syslogLogger struct {
@@ -41,25 +40,25 @@ type syslogHook struct {
 
 // enableSyslogLogger - enable logger at raddr.
 func enableSyslogLogger(raddr string) {
-	syslogHook, e := newSyslog("udp", raddr, syslog.LOG_ERR, "MINIO")
-	fatalIf(probe.NewError(e), "Unable to instantiate syslog.", nil)
+	syslogHook, err := newSyslog("udp", raddr, syslog.LOG_ERR, "MINIO")
+	fatalIf(err, "Unable to initialize syslog logger.")
 
 	log.Hooks.Add(syslogHook)               // Add syslog hook.
 	log.Formatter = &logrus.JSONFormatter{} // JSON formatted log.
-	log.Level = logrus.InfoLevel            // Minimum log level.
+	log.Level = logrus.ErrorLevel           // Minimum log level.
 }
 
 // newSyslog - Creates a hook to be added to an instance of logger.
 func newSyslog(network, raddr string, priority syslog.Priority, tag string) (*syslogHook, error) {
-	w, e := syslog.Dial(network, raddr, priority, tag)
-	return &syslogHook{w, network, raddr}, e
+	w, err := syslog.Dial(network, raddr, priority, tag)
+	return &syslogHook{w, network, raddr}, err
 }
 
 // Fire - fire the log event
 func (hook *syslogHook) Fire(entry *logrus.Entry) error {
-	line, e := entry.String()
-	if e != nil {
-		return fmt.Errorf("Unable to read entry, %v", e)
+	line, err := entry.String()
+	if err != nil {
+		return fmt.Errorf("Unable to read entry, %v", err)
 	}
 	switch entry.Level {
 	case logrus.PanicLevel:
@@ -68,12 +67,6 @@ func (hook *syslogHook) Fire(entry *logrus.Entry) error {
 		return hook.writer.Crit(line)
 	case logrus.ErrorLevel:
 		return hook.writer.Err(line)
-	case logrus.WarnLevel:
-		return hook.writer.Warning(line)
-	case logrus.InfoLevel:
-		return hook.writer.Info(line)
-	case logrus.DebugLevel:
-		return hook.writer.Debug(line)
 	default:
 		return nil
 	}
@@ -85,8 +78,5 @@ func (hook *syslogHook) Levels() []logrus.Level {
 		logrus.PanicLevel,
 		logrus.FatalLevel,
 		logrus.ErrorLevel,
-		logrus.WarnLevel,
-		logrus.InfoLevel,
-		logrus.DebugLevel,
 	}
 }

logger-syslog-hook_windows.go

@@ -18,8 +18,6 @@
 
 package main
 
-import "github.com/minio/minio/pkg/probe"
-
 type syslogLogger struct {
 	Enable bool   `json:"enable"`
 	Addr   string `json:"address"`
@@ -28,5 +26,5 @@ type syslogLogger struct {
 
 // enableSyslogLogger - unsupported on windows.
 func enableSyslogLogger(raddr string) {
-	fatalIf(probe.NewError(errSyslogNotSupported), "Unable to enable syslog.", nil)
+	fatalIf(errSyslogNotSupported, "Unable to enable syslog.")
 }

logger.go

@@ -17,11 +17,17 @@
 package main
 
 import (
-	"encoding/json"
+	"bufio"
+	"bytes"
+	"os"
 	"reflect"
+	"runtime"
+	"runtime/debug"
+	"strconv"
+	"strings"
 
 	"github.com/Sirupsen/logrus"
-	"github.com/minio/minio/pkg/probe"
+	"github.com/dustin/go-humanize"
 )
 
 type fields map[string]interface{}
@@ -42,40 +48,74 @@ type logger struct {
 	// Add new loggers here.
 }
 
+// sysInfo returns useful system statistics.
+func sysInfo() map[string]string {
+	host, err := os.Hostname()
+	if err != nil {
+		host = ""
+	}
+	memstats := &runtime.MemStats{}
+	runtime.ReadMemStats(memstats)
+	return map[string]string{
+		"host.name":      host,
+		"host.os":        runtime.GOOS,
+		"host.arch":      runtime.GOARCH,
+		"host.lang":      runtime.Version(),
+		"host.cpus":      strconv.Itoa(runtime.NumCPU()),
+		"mem.used":       humanize.Bytes(memstats.Alloc),
+		"mem.total":      humanize.Bytes(memstats.Sys),
+		"mem.heap.used":  humanize.Bytes(memstats.HeapAlloc),
+		"mem.heap.total": humanize.Bytes(memstats.HeapSys),
+	}
+}
+
+// stackInfo returns printable stack trace.
+func stackInfo() string {
+	// Convert stack-trace bytes to io.Reader.
+	rawStack := bufio.NewReader(bytes.NewBuffer(debug.Stack()))
+	// Skip stack trace lines until our real caller.
+	for i := 0; i <= 4; i++ {
+		rawStack.ReadLine()
+	}
+
+	// Read the rest of useful stack trace.
+	stackBuf := new(bytes.Buffer)
+	stackBuf.ReadFrom(rawStack)
+
+	// Strip GOPATH of the build system and return.
+	return strings.Replace(stackBuf.String(), minioGOPATH+"/src/", "", -1)
+}
+
 // errorIf synonymous with fatalIf but doesn't exit on error != nil
-func errorIf(err *probe.Error, msg string, fields map[string]interface{}) {
+func errorIf(err error, msg string, data ...interface{}) {
 	if err == nil {
 		return
 	}
-	if fields == nil {
-		fields = make(map[string]interface{})
+	sysInfo := sysInfo()
+	fields := logrus.Fields{
+		"cause":   err.Error(),
+		"type":    reflect.TypeOf(err),
+		"sysInfo": sysInfo,
 	}
-	fields["Error"] = struct {
-		Cause     string             `json:"cause,omitempty"`
-		Type      string             `json:"type,omitempty"`
-		CallTrace []probe.TracePoint `json:"trace,omitempty"`
-		SysInfo   map[string]string  `json:"sysinfo,omitempty"`
-	}{
-		err.Cause.Error(),
-		reflect.TypeOf(err.Cause).String(),
-		err.CallTrace,
-		err.SysInfo,
+	if globalTrace {
+		fields["stack"] = "\n" + stackInfo()
 	}
-	log.WithFields(fields).Error(msg)
+	log.WithFields(fields).Errorf(msg, data...)
 }
 
 // fatalIf wrapper function which takes error and prints jsonic error messages.
-func fatalIf(err *probe.Error, msg string, fields map[string]interface{}) {
+func fatalIf(err error, msg string, data ...interface{}) {
 	if err == nil {
 		return
 	}
-	if fields == nil {
-		fields = make(map[string]interface{})
+	sysInfo := sysInfo()
+	fields := logrus.Fields{
+		"cause":   err.Error(),
+		"type":    reflect.TypeOf(err),
+		"sysInfo": sysInfo,
 	}
-
-	fields["error"] = err.ToGoError()
-	if jsonErr, e := json.Marshal(err); e == nil {
-		fields["probe"] = string(jsonErr)
+	if globalTrace {
+		fields["stack"] = "\n" + stackInfo()
 	}
-	log.WithFields(fields).Fatal(msg)
+	log.WithFields(fields).Fatalf(msg, data...)
 }

logger_test.go

@@ -22,7 +22,6 @@ import (
 	"errors"
 
 	"github.com/Sirupsen/logrus"
-	"github.com/minio/minio/pkg/probe"
 
 	. "gopkg.in/check.v1"
 )
@@ -37,12 +36,12 @@ func (s *LoggerSuite) TestLogger(c *C) {
 	log.Out = &buffer
 	log.Formatter = new(logrus.JSONFormatter)
 
-	errorIf(probe.NewError(errors.New("Fake error")), "Failed with error.", nil)
+	errorIf(errors.New("Fake error"), "Failed with error.")
 	err := json.Unmarshal(buffer.Bytes(), &fields)
 	c.Assert(err, IsNil)
 	c.Assert(fields["level"], Equals, "error")
 
-	msg, ok := fields["Error"]
+	msg, ok := fields["cause"]
 	c.Assert(ok, Equals, true)
-	c.Assert(msg.(map[string]interface{})["cause"], Equals, "Fake error")
+	c.Assert(msg, Equals, "Fake error")
 }

main.go

@@ -19,15 +19,12 @@ package main
 import (
 	"fmt"
 	"os"
-	"runtime"
+	"path/filepath"
 	"sort"
-	"strconv"
 
-	"github.com/dustin/go-humanize"
 	"github.com/minio/cli"
 	"github.com/minio/mc/pkg/console"
-	"github.com/minio/minio/pkg/probe"
-	"github.com/olekukonko/ts"
+	"github.com/pkg/profile"
 )
 
 var (
@@ -58,16 +55,16 @@ FLAGS:
   {{end}}{{end}}
 VERSION:
   ` + minioVersion +
-	`{{ "\n"}}{{range $key, $value := ExtraInfo}}
-{{$key}}:
-  {{$value}}
-{{end}}`
+	`{{ "\n"}}`
 
 // init - check the environment before main starts
 func init() {
 	// Check if minio was compiled using a supported version of Golang.
 	checkGoVersion()
 
+	// Set global trace flag.
+	globalTrace = os.Getenv("MINIO_TRACE") == "1"
+
 	// It is an unsafe practice to run network services as
 	// root. Containers are an exception.
 	if !isContainerized() && os.Geteuid() == 0 {
@@ -85,36 +82,11 @@ func migrate() {
 func enableLoggers() {
 	// Enable all loggers here.
 	enableConsoleLogger()
+	enableFileLogger()
 
 	// Add your logger here.
 }
 
-// Tries to get os/arch/platform specific information
-// Returns a map of current os/arch/platform/memstats
-func getSystemData() map[string]string {
-	host, err := os.Hostname()
-	if err != nil {
-		host = ""
-	}
-	memstats := &runtime.MemStats{}
-	runtime.ReadMemStats(memstats)
-	mem := fmt.Sprintf("Used: %s | Allocated: %s | Used-Heap: %s | Allocated-Heap: %s",
-		humanize.Bytes(memstats.Alloc),
-		humanize.Bytes(memstats.TotalAlloc),
-		humanize.Bytes(memstats.HeapAlloc),
-		humanize.Bytes(memstats.HeapSys))
-	platform := fmt.Sprintf("Host: %s | OS: %s | Arch: %s",
-		host,
-		runtime.GOOS,
-		runtime.GOARCH)
-	goruntime := fmt.Sprintf("Version: %s | CPUs: %s", runtime.Version(), strconv.Itoa(runtime.NumCPU()))
-	return map[string]string{
-		"PLATFORM": platform,
-		"RUNTIME":  goruntime,
-		"MEM":      mem,
-	}
-}
-
 func findClosestCommands(command string) []string {
 	var closestCommands []string
 	for _, value := range commandsTree.PrefixMatch(command) {
@@ -146,8 +118,8 @@ func registerApp() *cli.App {
 	app := cli.NewApp()
 	app.Name = "Minio"
 	app.Author = "Minio.io"
-	app.Usage = "Distributed Object Storage Server for Micro Services."
-	app.Description = `Micro services environment provisions one Minio server per application instance. Scalability is achieved through large number of smaller personalized instances. This version of the Minio binary is built using Filesystem storage backend for magnetic and solid state disks.`
+	app.Usage = "Cloud Storage Server."
+	app.Description = `Minio is an Amazon S3 compatible object storage server. Use it to store photos, videos, VMs, containers, log files, or any blob of data as objects.`
 	app.Flags = append(minioFlags, globalFlags...)
 	app.Commands = commands
 	app.CustomAppHelpTemplate = minioHelpTemplate
@@ -175,16 +147,14 @@ func checkMainSyntax(c *cli.Context) {
 	}
 }
 
-func main() {
-	probe.Init() // Set project's root source path.
-	probe.SetAppInfo("Release-Tag", minioReleaseTag)
-	probe.SetAppInfo("Commit-ID", minioShortCommitID)
+// mustGetProfilePath must get location that the profile will be written to.
+func mustGetProfilePath() string {
+	return filepath.Join(mustGetConfigPath(), globalMinioProfilePath)
+}
 
+func main() {
 	app := registerApp()
 	app.Before = func(c *cli.Context) error {
-		// Set global flags.
-		setGlobalsFromContext(c)
-
 		// Sets new config folder.
 		setGlobalConfigPath(c.GlobalString("config-dir"))
 
@@ -196,11 +166,17 @@ func main() {
 
 		// Initialize config.
 		err := initConfig()
-		fatalIf(err.Trace(), "Unable to initialize minio config.", nil)
+		fatalIf(err, "Unable to initialize minio config.")
 
 		// Enable all loggers by now.
 		enableLoggers()
 
+		// Initialize name space lock.
+		initNSLock()
+
+		// Set global quiet flag.
+		globalQuiet = c.Bool("quiet") || c.GlobalBool("quiet")
+
 		// Do not print update messages, if quiet flag is set.
 		if !globalQuiet {
 			// Do not print any errors in release update function.
@@ -211,19 +187,21 @@ func main() {
 			}
 		}
 
-		// Return here.
 		return nil
 	}
-	app.ExtraInfo = func() map[string]string {
-		if _, e := ts.GetSize(); e != nil {
-			globalQuiet = true
-		}
-		// Enable if debug is enabled.
-		if globalDebug {
-			return getSystemData()
-		}
-		return make(map[string]string)
+
+	// Set ``MINIO_PROFILE_DIR`` to the directory where profiling information should be persisted
+	profileDir := os.Getenv("MINIO_PROFILE_DIR")
+	// Enable profiler if ``MINIO_PROFILER`` is set. Supported options are [cpu, mem, block].
+	switch os.Getenv("MINIO_PROFILER") {
+	case "cpu":
+		defer profile.Start(profile.CPUProfile, profile.ProfilePath(profileDir)).Stop()
+	case "mem":
+		defer profile.Start(profile.MemProfile, profile.ProfilePath(profileDir)).Stop()
+	case "block":
+		defer profile.Start(profile.BlockProfile, profile.ProfilePath(profileDir)).Stop()
 	}
+
 	// Run the app - exit on error.
 	app.RunAndExitOnError()
 }

namespace-lock.go

@@ -0,0 +1,128 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"errors"
+	"sync"
+)
+
+// nsParam - carries name space resource.
+type nsParam struct {
+	volume string
+	path   string
+}
+
+// nsLock - provides primitives for locking critical namespace regions.
+type nsLock struct {
+	*sync.RWMutex
+	ref uint
+}
+
+// nsLockMap - namespace lock map, provides primitives to Lock,
+// Unlock, RLock and RUnlock.
+type nsLockMap struct {
+	lockMap map[nsParam]*nsLock
+	mutex   *sync.Mutex
+}
+
+// Global name space lock.
+var nsMutex *nsLockMap
+
+// initNSLock - initialize name space lock map.
+func initNSLock() {
+	nsMutex = &nsLockMap{
+		lockMap: make(map[nsParam]*nsLock),
+		mutex:   &sync.Mutex{},
+	}
+}
+
+// Lock the namespace resource.
+func (n *nsLockMap) lock(volume, path string, readLock bool) {
+	n.mutex.Lock()
+
+	param := nsParam{volume, path}
+	nsLk, found := n.lockMap[param]
+	if !found {
+		nsLk = &nsLock{
+			RWMutex: &sync.RWMutex{},
+			ref:     0,
+		}
+		n.lockMap[param] = nsLk
+	}
+	nsLk.ref++ // Update ref count here to avoid multiple races.
+	// Unlock map before Locking NS which might block.
+	n.mutex.Unlock()
+
+	// Locking here can block.
+	if readLock {
+		nsLk.RLock()
+	} else {
+		nsLk.Lock()
+	}
+}
+
+// Unlock the namespace resource.
+func (n *nsLockMap) unlock(volume, path string, readLock bool) {
+	// nsLk.Unlock() will not block, hence locking the map for the entire function is fine.
+	n.mutex.Lock()
+	defer n.mutex.Unlock()
+
+	param := nsParam{volume, path}
+	if nsLk, found := n.lockMap[param]; found {
+		if readLock {
+			nsLk.RUnlock()
+		} else {
+			nsLk.Unlock()
+		}
+		if nsLk.ref == 0 {
+			errorIf(errors.New("Namespace reference count cannot be 0."), "Invalid reference count detected.")
+		}
+		if nsLk.ref != 0 {
+			nsLk.ref--
+		}
+		if nsLk.ref == 0 {
+			// Remove from the map if there are no more references.
+			delete(n.lockMap, param)
+		}
+	}
+}
+
+// Lock - locks the given resource for writes, using a previously
+// allocated name space lock or initializing a new one.
+func (n *nsLockMap) Lock(volume, path string) {
+	readLock := false
+	n.lock(volume, path, readLock)
+}
+
+// Unlock - unlocks any previously acquired write locks.
+func (n *nsLockMap) Unlock(volume, path string) {
+	readLock := false
+	n.unlock(volume, path, readLock)
+}
+
+// RLock - locks any previously acquired read locks.
+func (n *nsLockMap) RLock(volume, path string) {
+	readLock := true
+	n.lock(volume, path, readLock)
+}
+
+// RUnlock - unlocks any previously acquired read locks.
+func (n *nsLockMap) RUnlock(volume, path string) {
+	readLock := true
+	n.unlock(volume, path, readLock)
+}

notifier.go

@@ -22,12 +22,11 @@ import (
 	"strings"
 
 	"github.com/fatih/color"
-	"github.com/minio/minio/pkg/probe"
 	"github.com/olekukonko/ts"
 )
 
 // colorizeUpdateMessage - inspired from Yeoman project npm package https://github.com/yeoman/update-notifier
-func colorizeUpdateMessage(updateString string) (string, *probe.Error) {
+func colorizeUpdateMessage(updateString string) (string, error) {
 	// Initialize coloring.
 	cyan := color.New(color.FgCyan, color.Bold).SprintFunc()
 	yellow := color.New(color.FgYellow, color.Bold).SprintfFunc()
@@ -47,7 +46,7 @@ func colorizeUpdateMessage(updateString string) (string, *probe.Error) {
 
 	terminal, err := ts.GetSize()
 	if err != nil {
-		return "", probe.NewError(err)
+		return "", err
 	}
 
 	var message string

object-api-getobjectinfo_test.go

@@ -0,0 +1,156 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"bytes"
+	"crypto/md5"
+	"encoding/hex"
+	"io/ioutil"
+	"os"
+	"strconv"
+	"testing"
+)
+
+// Wrapper for calling GetObjectInfo tests for both XL multiple disks and single node setup.
+func TestGetObjectInfo(t *testing.T) {
+	ExecObjectLayerTest(t, testGetObjectInfo)
+}
+
+// Testing GetObjectInfo().
+func testGetObjectInfo(obj ObjectLayer, instanceType string, t *testing.T) {
+	// This bucket is used for testing getObjectInfo operations.
+	err := obj.MakeBucket("test-getobjectinfo")
+	if err != nil {
+		t.Fatalf("%s : %s", instanceType, err.Error())
+	}
+	_, err = obj.PutObject("test-getobjectinfo", "Asia/asiapics.jpg", int64(len("asiapics")), bytes.NewBufferString("asiapics"), nil)
+	if err != nil {
+		t.Fatalf("%s : %s", instanceType, err.Error())
+	}
+	resultCases := []ObjectInfo{
+		// ObjectInfo -1.
+		// ObjectName set to a existing object in the test case (Test case 14).
+		{Bucket: "test-getobjectinfo", Name: "Asia/asiapics.jpg", ContentType: "image/jpeg", IsDir: false},
+	}
+	testCases := []struct {
+		bucketName string
+		objectName string
+
+		// Expected output of GetObjectInfo.
+		result ObjectInfo
+		err    error
+		// Flag indicating whether the test is expected to pass or not.
+		shouldPass bool
+	}{
+		// Test cases with invalid bucket names ( Test number 1-4 ).
+		{".test", "", ObjectInfo{}, BucketNameInvalid{Bucket: ".test"}, false},
+		{"Test", "", ObjectInfo{}, BucketNameInvalid{Bucket: "Test"}, false},
+		{"---", "", ObjectInfo{}, BucketNameInvalid{Bucket: "---"}, false},
+		{"ad", "", ObjectInfo{}, BucketNameInvalid{Bucket: "ad"}, false},
+		// Test cases with valid but non-existing bucket names (Test number 5-7).
+		{"abcdefgh", "abc", ObjectInfo{}, BucketNotFound{Bucket: "abcdefgh"}, false},
+		{"ijklmnop", "efg", ObjectInfo{}, BucketNotFound{Bucket: "ijklmnop"}, false},
+		// Test cases with valid but non-existing bucket names and invalid object name (Test number 8-9).
+		{"test-getobjectinfo", "", ObjectInfo{}, ObjectNameInvalid{Bucket: "test-getobjectinfo", Object: ""}, false},
+		{"test-getobjectinfo", "", ObjectInfo{}, ObjectNameInvalid{Bucket: "test-getobjectinfo", Object: ""}, false},
+		// Test cases with non-existing object name with existing bucket (Test number 10-12).
+		{"test-getobjectinfo", "Africa", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Africa"}, false},
+		{"test-getobjectinfo", "Antartica", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Antartica"}, false},
+		{"test-getobjectinfo", "Asia/myfile", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Asia/myfile"}, false},
+		// Test case with existing bucket but object name set to a directory (Test number 13).
+		{"test-getobjectinfo", "Asia", ObjectInfo{}, ObjectNotFound{Bucket: "test-getobjectinfo", Object: "Asia"}, false},
+		// Valid case with existing object (Test number 14).
+		{"test-getobjectinfo", "Asia/asiapics.jpg", resultCases[0], nil, true},
+	}
+	for i, testCase := range testCases {
+		result, err := obj.GetObjectInfo(testCase.bucketName, testCase.objectName)
+		if err != nil && testCase.shouldPass {
+			t.Errorf("Test %d: %s: Expected to pass, but failed with: <ERROR> %s", i+1, instanceType, err.Error())
+		}
+		if err == nil && !testCase.shouldPass {
+			t.Errorf("Test %d: %s: Expected to fail with <ERROR> \"%s\", but passed instead", i+1, instanceType, testCase.err.Error())
+		}
+		// Failed as expected, but does it fail for the expected reason.
+		if err != nil && !testCase.shouldPass {
+			if testCase.err.Error() != err.Error() {
+				t.Errorf("Test %d: %s: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead", i+1, instanceType, testCase.err.Error(), err.Error())
+			}
+		}
+
+		// Test passes as expected, but the output values are verified for correctness here.
+		if err == nil && testCase.shouldPass {
+			if testCase.result.Bucket != result.Bucket {
+				t.Fatalf("Test %d: %s: Expected Bucket name to be '%s', but found '%s' instead", i+1, instanceType, testCase.result.Bucket, result.Bucket)
+			}
+			if testCase.result.Name != result.Name {
+				t.Errorf("Test %d: %s: Expected Object name to be %s, but instead found it to be %s", i+1, instanceType, testCase.result.Name, result.Name)
+			}
+			if testCase.result.ContentType != result.ContentType {
+				t.Errorf("Test %d: %s: Expected Content Type of the object to be %v, but instead found it to be %v", i+1, instanceType, testCase.result.ContentType, result.ContentType)
+			}
+			if testCase.result.IsDir != result.IsDir {
+				t.Errorf("Test %d: %s: Expected IsDir flag of the object to be %v, but instead found it to be %v", i+1, instanceType, testCase.result.IsDir, result.IsDir)
+			}
+		}
+	}
+}
+
+func BenchmarkGetObjectFS(b *testing.B) {
+	// Make a temporary directory to use as the obj.
+	directory, err := ioutil.TempDir("", "minio-benchmark-getobject")
+	if err != nil {
+		b.Fatal(err)
+	}
+	defer os.RemoveAll(directory)
+
+	// Create the obj.
+	obj, err := newFSObjects(directory)
+	if err != nil {
+		b.Fatal(err)
+	}
+
+	// Make a bucket and put in a few objects.
+	err = obj.MakeBucket("bucket")
+	if err != nil {
+		b.Fatal(err)
+	}
+
+	text := "Jack and Jill went up the hill / To fetch a pail of water."
+	hasher := md5.New()
+	hasher.Write([]byte(text))
+	metadata := make(map[string]string)
+	for i := 0; i < 10; i++ {
+		metadata["md5Sum"] = hex.EncodeToString(hasher.Sum(nil))
+		_, err = obj.PutObject("bucket", "object"+strconv.Itoa(i), int64(len(text)), bytes.NewBufferString(text), metadata)
+		if err != nil {
+			b.Fatal(err)
+		}
+	}
+
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		var buffer = new(bytes.Buffer)
+		err = obj.GetObject("bucket", "object"+strconv.Itoa(i%10), 0, int64(len([]byte(text))), buffer)
+		if err != nil {
+			b.Error(err)
+		}
+		if buffer.Len() != len(text) {
+			b.Errorf("GetObject returned incorrect length %d (should be %d)\n", buffer.Len(), len(text))
+		}
+	}
+}

object-api-interface.go

@@ -1,33 +0,0 @@
-package main
-
-import (
-	"io"
-
-	"github.com/minio/minio/pkg/probe"
-)
-
-// ObjectAPI interface.
-type ObjectAPI interface {
-	// Bucket resource API.
-	DeleteBucket(bucket string) *probe.Error
-	ListBuckets() ([]BucketInfo, *probe.Error)
-	MakeBucket(bucket string) *probe.Error
-	GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
-
-	// Bucket query API.
-	ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
-	ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
-
-	// Object resource API.
-	GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
-	GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
-	PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
-	DeleteObject(bucket, object string) *probe.Error
-
-	// Object query API.
-	NewMultipartUpload(bucket, object string) (string, *probe.Error)
-	PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
-	ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
-	CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
-	AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
-}

object-api-listobjects_test.go

@@ -26,69 +26,64 @@ import (
 	"testing"
 )
 
+// Wrapper for calling ListObjects tests for both XL multiple disks and single node setup.
 func TestListObjects(t *testing.T) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-list-object-test")
-	if e != nil {
-		t.Fatal(e)
-	}
-	defer os.RemoveAll(directory)
+	ExecObjectLayerTest(t, testListObjects)
+}
+
+// Unit test for ListObjects in general.
+func testListObjects(obj ObjectLayer, instanceType string, t *testing.T) {
 
-	// Create the fs.
-	fs, err := newFS(directory)
-	if err != nil {
-		t.Fatal(err)
-	}
 	// This bucket is used for testing ListObject operations.
-	err = fs.MakeBucket("test-bucket-list-object")
+	err := obj.MakeBucket("test-bucket-list-object")
 	if err != nil {
-		t.Fatal(err)
+		t.Fatalf("%s : %s", instanceType, err.Error())
 	}
 	// Will not store any objects in this bucket,
 	// Its to test ListObjects on an empty bucket.
-	err = fs.MakeBucket("empty-bucket")
+	err = obj.MakeBucket("empty-bucket")
 	if err != nil {
-		t.Fatal(err)
+		t.Fatalf("%s : %s", instanceType, err.Error())
 	}
 
-	tmpfile, e := ioutil.TempFile("", "simple-file.txt")
-	if e != nil {
-		t.Fatal(e)
+	tmpfile, err := ioutil.TempFile("", "simple-file.txt")
+	if err != nil {
+		t.Fatalf("%s : %s", instanceType, err.Error())
 	}
 	defer os.Remove(tmpfile.Name()) // clean up
 
-	_, err = fs.PutObject("test-bucket-list-object", "Asia-maps", int64(len("asia-maps")), bytes.NewBufferString("asia-maps"), nil)
+	_, err = obj.PutObject("test-bucket-list-object", "Asia-maps", int64(len("asia-maps")), bytes.NewBufferString("asia-maps"), nil)
 	if err != nil {
-		t.Fatal(e)
+		t.Fatalf("%s : %s", instanceType, err.Error())
 	}
 
-	_, err = fs.PutObject("test-bucket-list-object", "Asia/India/India-summer-photos-1", int64(len("contentstring")), bytes.NewBufferString("contentstring"), nil)
+	_, err = obj.PutObject("test-bucket-list-object", "Asia/India/India-summer-photos-1", int64(len("contentstring")), bytes.NewBufferString("contentstring"), nil)
 	if err != nil {
-		t.Fatal(e)
+		t.Fatalf("%s : %s", instanceType, err.Error())
 	}
 
-	_, err = fs.PutObject("test-bucket-list-object", "Asia/India/Karnataka/Bangalore/Koramangala/pics", int64(len("contentstring")), bytes.NewBufferString("contentstring"), nil)
+	_, err = obj.PutObject("test-bucket-list-object", "Asia/India/Karnataka/Bangalore/Koramangala/pics", int64(len("contentstring")), bytes.NewBufferString("contentstring"), nil)
 	if err != nil {
-		t.Fatal(e)
+		t.Fatalf("%s : %s", instanceType, err.Error())
 	}
 
 	for i := 0; i < 2; i++ {
 		key := "newPrefix" + strconv.Itoa(i)
-		_, err = fs.PutObject("test-bucket-list-object", key, int64(len(key)), bytes.NewBufferString(key), nil)
+		_, err = obj.PutObject("test-bucket-list-object", key, int64(len(key)), bytes.NewBufferString(key), nil)
 		if err != nil {
-			t.Fatal(err)
+			t.Fatalf("%s : %s", instanceType, err.Error())
 		}
 	}
-	_, err = fs.PutObject("test-bucket-list-object", "newzen/zen/recurse/again/again/again/pics", int64(len("recurse")), bytes.NewBufferString("recurse"), nil)
+	_, err = obj.PutObject("test-bucket-list-object", "newzen/zen/recurse/again/again/again/pics", int64(len("recurse")), bytes.NewBufferString("recurse"), nil)
 	if err != nil {
-		t.Fatal(e)
+		t.Fatalf("%s : %s", instanceType, err.Error())
 	}
 
 	for i := 0; i < 3; i++ {
 		key := "obj" + strconv.Itoa(i)
-		_, err = fs.PutObject("test-bucket-list-object", key, int64(len(key)), bytes.NewBufferString(key), nil)
+		_, err = obj.PutObject("test-bucket-list-object", key, int64(len(key)), bytes.NewBufferString(key), nil)
 		if err != nil {
-			t.Fatal(err)
+			t.Fatalf("%s : %s", instanceType, err.Error())
 		}
 	}
 
@@ -418,6 +413,12 @@ func TestListObjects(t *testing.T) {
 				{Name: "obj2"},
 			},
 		},
+		// ListObjectsResult-30.
+		// Prefix and Delimiter is set to '/', (testCase 62).
+		{
+			IsTruncated: false,
+			Objects:     []ObjectInfo{},
+		},
 	}
 
 	testCases := []struct {
@@ -526,37 +527,43 @@ func TestListObjects(t *testing.T) {
 		// Test with marker set as hierarhical value and with delimiter. (60-61)
 		{"test-bucket-list-object", "", "Asia/India/India-summer-photos-1", "/", 10, resultCases[28], nil, true},
 		{"test-bucket-list-object", "", "Asia/India/Karnataka/Bangalore/Koramangala/pics", "/", 10, resultCases[29], nil, true},
+		// Test with prefix and delimiter set to '/'. (62)
+		{"test-bucket-list-object", "/", "", "/", 10, resultCases[30], nil, true},
 	}
 
 	for i, testCase := range testCases {
-		result, err := fs.ListObjects(testCase.bucketName, testCase.prefix, testCase.marker, testCase.delimeter, testCase.maxKeys)
+		result, err := obj.ListObjects(testCase.bucketName, testCase.prefix, testCase.marker, testCase.delimeter, testCase.maxKeys)
 		if err != nil && testCase.shouldPass {
-			t.Errorf("Test %d: Expected to pass, but failed with: <ERROR> %s", i+1, err.Cause.Error())
+			t.Errorf("Test %d: %s:  Expected to pass, but failed with: <ERROR> %s", i+1, instanceType, err.Error())
 		}
 		if err == nil && !testCase.shouldPass {
-			t.Errorf("Test %d: Expected to fail with <ERROR> \"%s\", but passed instead", i+1, testCase.err.Error())
+			t.Errorf("Test %d: %s: Expected to fail with <ERROR> \"%s\", but passed instead", i+1, instanceType, testCase.err.Error())
 		}
 		// Failed as expected, but does it fail for the expected reason.
 		if err != nil && !testCase.shouldPass {
-			if !strings.Contains(err.Cause.Error(), testCase.err.Error()) {
-				t.Errorf("Test %d: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead", i+1, testCase.err.Error(), err.Cause.Error())
+			if !strings.Contains(err.Error(), testCase.err.Error()) {
+				t.Errorf("Test %d: %s: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead", i+1, instanceType, testCase.err.Error(), err.Error())
 			}
 		}
-		// Since there are cases for which ListObjects fails, this is necessary.
-		// Test passes as expected, but the output values are verified for correctness here.
+		// Since there are cases for which ListObjects fails, this is
+		// necessary. Test passes as expected, but the output values
+		// are verified for correctness here.
 		if err == nil && testCase.shouldPass {
-			// The length of the expected ListObjectsResult.Objects should match in both expected result from test cases and in the output.
-			// On failure calling t.Fatalf, otherwise it may lead to index out of range error in assertion following this.
+			// The length of the expected ListObjectsResult.Objects
+			// should match in both expected result from test cases
+			// and in the output. On failure calling t.Fatalf,
+			// otherwise it may lead to index out of range error in
+			// assertion following this.
 			if len(testCase.result.Objects) != len(result.Objects) {
-				t.Fatalf("Test %d: Expected number of object in the result to be '%d', but found '%d' objects instead", i+1, len(testCase.result.Objects), len(result.Objects))
+				t.Fatalf("Test %d: %s: Expected number of object in the result to be '%d', but found '%d' objects instead", i+1, instanceType, len(testCase.result.Objects), len(result.Objects))
 			}
 			for j := 0; j < len(testCase.result.Objects); j++ {
 				if testCase.result.Objects[j].Name != result.Objects[j].Name {
-					t.Errorf("Test %d: Expected object name to be \"%s\", but found \"%s\" instead", i+1, testCase.result.Objects[j].Name, result.Objects[j].Name)
+					t.Errorf("Test %d: %s: Expected object name to be \"%s\", but found \"%s\" instead", i+1, instanceType, testCase.result.Objects[j].Name, result.Objects[j].Name)
 				}
 			}
 			if testCase.result.IsTruncated != result.IsTruncated {
-				t.Errorf("Test %d: Expected IsTruncated flag to be %v, but instead found it to be %v", i+1, testCase.result.IsTruncated, result.IsTruncated)
+				t.Errorf("Test %d: %s: Expected IsTruncated flag to be %v, but instead found it to be %v", i+1, instanceType, testCase.result.IsTruncated, result.IsTruncated)
 			}
 
 		}
@@ -565,28 +572,28 @@ func TestListObjects(t *testing.T) {
 }
 
 func BenchmarkListObjects(b *testing.B) {
-	// Make a temporary directory to use as the fs.
-	directory, e := ioutil.TempDir("", "minio-list-benchmark")
-	if e != nil {
-		b.Fatal(e)
+	// Make a temporary directory to use as the obj.
+	directory, err := ioutil.TempDir("", "minio-list-benchmark")
+	if err != nil {
+		b.Fatal(err)
 	}
 	defer os.RemoveAll(directory)
 
-	// Create the fs.
-	fs, err := newFS(directory)
+	// Create the obj.
+	obj, err := newFSObjects(directory)
 	if err != nil {
 		b.Fatal(err)
 	}
 
 	// Create a bucket.
-	err = fs.MakeBucket("ls-benchmark-bucket")
+	err = obj.MakeBucket("ls-benchmark-bucket")
 	if err != nil {
 		b.Fatal(err)
 	}
 
 	for i := 0; i < 20000; i++ {
 		key := "obj" + strconv.Itoa(i)
-		_, err = fs.PutObject("ls-benchmark-bucket", key, int64(len(key)), bytes.NewBufferString(key), nil)
+		_, err = obj.PutObject("ls-benchmark-bucket", key, int64(len(key)), bytes.NewBufferString(key), nil)
 		if err != nil {
 			b.Fatal(err)
 		}
@@ -596,7 +603,7 @@ func BenchmarkListObjects(b *testing.B) {
 
 	// List the buckets over and over and over.
 	for i := 0; i < b.N; i++ {
-		_, err = fs.ListObjects("ls-benchmark-bucket", "", "obj9000", "", -1)
+		_, err = obj.ListObjects("ls-benchmark-bucket", "", "obj9000", "", -1)
 		if err != nil {
 			b.Fatal(err)
 		}

object-api_test.go

@@ -1,5 +1,5 @@
 /*
- * Minio Cloud Storage, (C) 2015 Minio, Inc.
+ * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -23,15 +23,45 @@ import (
 	. "gopkg.in/check.v1"
 )
 
-func (s *MyAPISuite) TestAPISuite(c *C) {
+type MySuite struct{}
+
+var _ = Suite(&MySuite{})
+
+func (s *MySuite) TestFSAPISuite(c *C) {
 	var storageList []string
-	create := func() ObjectAPI {
-		path, e := ioutil.TempDir(os.TempDir(), "minio-")
-		c.Check(e, IsNil)
-		storageList = append(storageList, path)
-		store, err := newFS(path)
+
+	// Initialize name space lock.
+	initNSLock()
+
+	create := func() ObjectLayer {
+		path, err := ioutil.TempDir(os.TempDir(), "minio-")
 		c.Check(err, IsNil)
-		return store
+		objAPI, err := newFSObjects(path)
+		c.Check(err, IsNil)
+		storageList = append(storageList, path)
+		return objAPI
+	}
+	APITestSuite(c, create)
+	defer removeRoots(c, storageList)
+}
+
+func (s *MySuite) TestXLAPISuite(c *C) {
+	var storageList []string
+
+	// Initialize name space lock.
+	initNSLock()
+
+	create := func() ObjectLayer {
+		var nDisks = 16 // Maximum disks.
+		var erasureDisks []string
+		for i := 0; i < nDisks; i++ {
+			path, err := ioutil.TempDir(os.TempDir(), "minio-")
+			c.Check(err, IsNil)
+			erasureDisks = append(erasureDisks, path)
+		}
+		objAPI, err := newXLObjects(erasureDisks)
+		c.Check(err, IsNil)
+		return objAPI
 	}
 	APITestSuite(c, create)
 	defer removeRoots(c, storageList)

object-common.go

@@ -0,0 +1,137 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"path/filepath"
+	"strings"
+	"sync"
+)
+
+const (
+	// Block size used for all internal operations version 1.
+	blockSizeV1 = 10 * 1024 * 1024 // 10MiB.
+)
+
+// House keeping code needed for FS.
+func fsHouseKeeping(storageDisk StorageAPI) error {
+	// Attempt to create `.minio`.
+	err := storageDisk.MakeVol(minioMetaBucket)
+	if err != nil {
+		if err != errVolumeExists && err != errDiskNotFound {
+			return err
+		}
+	}
+	// Cleanup all temp entries upon start.
+	err = cleanupDir(storageDisk, minioMetaBucket, tmpMetaPrefix)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// Depending on the disk type network or local, initialize storage API.
+func newStorageAPI(disk string) (storage StorageAPI, err error) {
+	if !strings.ContainsRune(disk, ':') || filepath.VolumeName(disk) != "" {
+		// Initialize filesystem storage API.
+		return newPosix(disk)
+	}
+	// Initialize rpc client storage API.
+	return newRPCClient(disk)
+}
+
+// House keeping code needed for XL.
+func xlHouseKeeping(storageDisks []StorageAPI) error {
+	// This happens for the first time, but keep this here since this
+	// is the only place where it can be made expensive optimizing all
+	// other calls. Create minio meta volume, if it doesn't exist yet.
+	var wg = &sync.WaitGroup{}
+
+	// Initialize errs to collect errors inside go-routine.
+	var errs = make([]error, len(storageDisks))
+
+	// Initialize all disks in parallel.
+	for index, disk := range storageDisks {
+		if disk == nil {
+			errs[index] = errDiskNotFound
+			continue
+		}
+		wg.Add(1)
+		go func(index int, disk StorageAPI) {
+			// Indicate this wait group is done.
+			defer wg.Done()
+
+			// Attempt to create `.minio`.
+			err := disk.MakeVol(minioMetaBucket)
+			if err != nil && err != errVolumeExists && err != errDiskNotFound {
+				errs[index] = err
+				return
+			}
+			// Cleanup all temp entries upon start.
+			err = cleanupDir(disk, minioMetaBucket, tmpMetaPrefix)
+			if err != nil {
+				errs[index] = err
+				return
+			}
+			errs[index] = nil
+		}(index, disk)
+	}
+
+	// Wait for all cleanup to finish.
+	wg.Wait()
+
+	// Return upon first error.
+	for _, err := range errs {
+		if err == nil {
+			continue
+		}
+		return toObjectErr(err, minioMetaBucket, tmpMetaPrefix)
+	}
+
+	// Return success here.
+	return nil
+}
+
+// Cleanup a directory recursively.
+func cleanupDir(storage StorageAPI, volume, dirPath string) error {
+	var delFunc func(string) error
+	// Function to delete entries recursively.
+	delFunc = func(entryPath string) error {
+		if !strings.HasSuffix(entryPath, slashSeparator) {
+			// No trailing "/" means that this is a file which can be deleted.
+			return storage.DeleteFile(volume, entryPath)
+		}
+		// If it's a directory, list and call delFunc() for each entry.
+		entries, err := storage.ListDir(volume, entryPath)
+		if err != nil {
+			if err == errFileNotFound {
+				// if dirPath prefix never existed.
+				return nil
+			}
+			return err
+		}
+		for _, entry := range entries {
+			err = delFunc(pathJoin(entryPath, entry))
+			if err != nil {
+				return err
+			}
+		}
+		return nil
+	}
+	err := delFunc(retainSlash(pathJoin(dirPath)))
+	return err
+}

object-datatypes.go

@@ -18,6 +18,12 @@ package main
 
 import "time"
 
+// StorageInfo - represents total capacity of underlying storage.
+type StorageInfo struct {
+	Total int64 // Total disk space.
+	Free  int64 // Free total available disk space.
+}
+
 // BucketInfo - bucket name and create date
 type BucketInfo struct {
 	Name    string
@@ -26,14 +32,14 @@ type BucketInfo struct {
 
 // ObjectInfo - object info.
 type ObjectInfo struct {
-	Bucket       string
-	Name         string
-	ModifiedTime time.Time
-	ContentType  string
-	MD5Sum       string
-	Size         int64
-	IsDir        bool
-	Err          error
+	Bucket          string
+	Name            string
+	ModTime         time.Time
+	Size            int64
+	IsDir           bool
+	MD5Sum          string
+	ContentType     string
+	ContentEncoding string
 }
 
 // ListPartsInfo - various types of object resources.

object-errors.go

@@ -0,0 +1,258 @@
+/*
+ * Minio Cloud Storage, (C) 2015 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"fmt"
+	"io"
+)
+
+// Converts underlying storage error. Convenience function written to
+// handle all cases where we have known types of errors returned by
+// underlying storage layer.
+func toObjectErr(err error, params ...string) error {
+	switch err {
+	case errVolumeNotFound:
+		if len(params) >= 1 {
+			return BucketNotFound{Bucket: params[0]}
+		}
+	case errVolumeNotEmpty:
+		if len(params) >= 1 {
+			return BucketNotEmpty{Bucket: params[0]}
+		}
+	case errVolumeExists:
+		if len(params) >= 1 {
+			return BucketExists{Bucket: params[0]}
+		}
+	case errDiskFull:
+		return StorageFull{}
+	case errIsNotRegular, errFileAccessDenied:
+		if len(params) >= 2 {
+			return ObjectExistsAsDirectory{
+				Bucket: params[0],
+				Object: params[1],
+			}
+		}
+	case errFileNotFound:
+		if len(params) >= 2 {
+			return ObjectNotFound{
+				Bucket: params[0],
+				Object: params[1],
+			}
+		}
+	case errFileNameTooLong:
+		if len(params) >= 2 {
+			return ObjectNameInvalid{
+				Bucket: params[0],
+				Object: params[1],
+			}
+		}
+	case errXLReadQuorum:
+		return InsufficientReadQuorum{}
+	case errXLWriteQuorum:
+		return InsufficientWriteQuorum{}
+	case io.ErrUnexpectedEOF, io.ErrShortWrite:
+		return IncompleteBody{}
+	}
+	return err
+}
+
+// StorageFull storage ran out of space.
+type StorageFull struct{}
+
+func (e StorageFull) Error() string {
+	return "Storage reached its minimum free disk threshold."
+}
+
+// InsufficientReadQuorum storage cannot satisfy quorum for read operation.
+type InsufficientReadQuorum struct{}
+
+func (e InsufficientReadQuorum) Error() string {
+	return "Storage resources are insufficient for the read operation."
+}
+
+// InsufficientWriteQuorum storage cannot satisfy quorum for write operation.
+type InsufficientWriteQuorum struct{}
+
+func (e InsufficientWriteQuorum) Error() string {
+	return "Storage resources are insufficient for the write operation."
+}
+
+// GenericError - generic object layer error.
+type GenericError struct {
+	Bucket string
+	Object string
+}
+
+// BucketNotFound bucket does not exist.
+type BucketNotFound GenericError
+
+func (e BucketNotFound) Error() string {
+	return "Bucket not found: " + e.Bucket
+}
+
+// BucketNotEmpty bucket is not empty.
+type BucketNotEmpty GenericError
+
+func (e BucketNotEmpty) Error() string {
+	return "Bucket not empty: " + e.Bucket
+}
+
+// ObjectNotFound object does not exist.
+type ObjectNotFound GenericError
+
+func (e ObjectNotFound) Error() string {
+	return "Object not found: " + e.Bucket + "#" + e.Object
+}
+
+// ObjectExistsAsDirectory object already exists as a directory.
+type ObjectExistsAsDirectory GenericError
+
+func (e ObjectExistsAsDirectory) Error() string {
+	return "Object exists on : " + e.Bucket + " as directory " + e.Object
+}
+
+// BucketExists bucket exists.
+type BucketExists GenericError
+
+func (e BucketExists) Error() string {
+	return "Bucket exists: " + e.Bucket
+}
+
+// BadDigest - Content-MD5 you specified did not match what we received.
+type BadDigest struct {
+	ExpectedMD5   string
+	CalculatedMD5 string
+}
+
+func (e BadDigest) Error() string {
+	return "Bad digest: Expected " + e.ExpectedMD5 + " is not valid with what we calculated " + e.CalculatedMD5
+}
+
+// UnsupportedDelimiter - unsupported delimiter.
+type UnsupportedDelimiter struct {
+	Delimiter string
+}
+
+func (e UnsupportedDelimiter) Error() string {
+	return fmt.Sprintf("delimiter '%s' is not supported. Only '/' is supported", e.Delimiter)
+}
+
+// InvalidUploadIDKeyCombination - invalid upload id and key marker combination.
+type InvalidUploadIDKeyCombination struct {
+	UploadIDMarker, KeyMarker string
+}
+
+func (e InvalidUploadIDKeyCombination) Error() string {
+	return fmt.Sprintf("Invalid combination of uploadID marker '%s' and marker '%s'", e.UploadIDMarker, e.KeyMarker)
+}
+
+// InvalidMarkerPrefixCombination - invalid marker and prefix combination.
+type InvalidMarkerPrefixCombination struct {
+	Marker, Prefix string
+}
+
+func (e InvalidMarkerPrefixCombination) Error() string {
+	return fmt.Sprintf("Invalid combination of marker '%s' and prefix '%s'", e.Marker, e.Prefix)
+}
+
+// BucketPolicyNotFound - no bucket policy found.
+type BucketPolicyNotFound GenericError
+
+func (e BucketPolicyNotFound) Error() string {
+	return "No bucket policy found for bucket: " + e.Bucket
+}
+
+/// Bucket related errors.
+
+// BucketNameInvalid - bucketname provided is invalid.
+type BucketNameInvalid GenericError
+
+// Return string an error formatted as the given text.
+func (e BucketNameInvalid) Error() string {
+	return "Bucket name invalid: " + e.Bucket
+}
+
+/// Object related errors.
+
+// ObjectNameInvalid - object name provided is invalid.
+type ObjectNameInvalid GenericError
+
+// Return string an error formatted as the given text.
+func (e ObjectNameInvalid) Error() string {
+	return "Object name invalid: " + e.Bucket + "#" + e.Object
+}
+
+// UnExpectedDataSize - Reader contains more/less data than specified.
+type UnExpectedDataSize struct {
+	Size int
+}
+
+func (e UnExpectedDataSize) Error() string {
+	return fmt.Sprintf("Contains more data than specified size of %d bytes.", e.Size)
+}
+
+// IncompleteBody You did not provide the number of bytes specified by the Content-Length HTTP header.
+type IncompleteBody GenericError
+
+// Return string an error formatted as the given text.
+func (e IncompleteBody) Error() string {
+	return e.Bucket + "#" + e.Object + "has incomplete body"
+}
+
+/// Multipart related errors.
+
+// MalformedUploadID malformed upload id.
+type MalformedUploadID struct {
+	UploadID string
+}
+
+func (e MalformedUploadID) Error() string {
+	return "Malformed upload id " + e.UploadID
+}
+
+// InvalidUploadID invalid upload id.
+type InvalidUploadID struct {
+	UploadID string
+}
+
+func (e InvalidUploadID) Error() string {
+	return "Invalid upload id " + e.UploadID
+}
+
+// InvalidPart One or more of the specified parts could not be found
+type InvalidPart struct{}
+
+func (e InvalidPart) Error() string {
+	return "One or more of the specified parts could not be found"
+}
+
+// InvalidPartOrder parts are not ordered as Requested
+type InvalidPartOrder struct {
+	UploadID string
+}
+
+func (e InvalidPartOrder) Error() string {
+	return "Invalid part order sent for " + e.UploadID
+}
+
+// PartTooSmall - error if part size is less than 5MB.
+type PartTooSmall struct{}
+
+func (e PartTooSmall) Error() string {
+	return "Part size should be atleast 5MB"
+}

object-handlers.go

@@ -17,6 +17,7 @@
 package main
 
 import (
+	"crypto/sha256"
 	"encoding/hex"
 	"encoding/xml"
 	"fmt"
@@ -27,12 +28,10 @@ import (
 	"sort"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 
-	fastSha256 "github.com/minio/minio/pkg/crypto/sha256"
-
 	mux "github.com/gorilla/mux"
-	"github.com/minio/minio/pkg/probe"
 )
 
 // supportedGetReqParams - supported request parameters for GET presigned request.
@@ -66,7 +65,6 @@ func errAllowableObjectNotFound(bucket string, r *http.Request) APIErrorCode {
 			return ErrAccessDenied
 		}
 	}
-
 	return ErrNoSuchKey
 }
 
@@ -74,7 +72,7 @@ func errAllowableObjectNotFound(bucket string, r *http.Request) APIErrorCode {
 // ----------
 // This implementation of the GET operation retrieves object. To use GET,
 // you must have READ access to the object.
-func (api objectStorageAPI) GetObjectHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) GetObjectHandler(w http.ResponseWriter, r *http.Request) {
 	var object, bucket string
 	vars := mux.Vars(r)
 	bucket = vars["bucket"]
@@ -97,27 +95,20 @@ func (api objectStorageAPI) GetObjectHandler(w http.ResponseWriter, r *http.Requ
 			return
 		}
 	}
-
+	// Fetch object stat info.
 	objInfo, err := api.ObjectAPI.GetObjectInfo(bucket, object)
 	if err != nil {
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, errAllowableObjectNotFound(bucket, r), r.URL.Path)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		default:
-			errorIf(err.Trace(), "GetObjectInfo failed.", nil)
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
+		errorIf(err, "Unable to fetch object info.")
+		apiErr := toAPIErrorCode(err)
+		if apiErr == ErrNoSuchKey {
+			apiErr = errAllowableObjectNotFound(bucket, r)
 		}
+		writeErrorResponse(w, r, apiErr, r.URL.Path)
 		return
 	}
 
 	// Verify 'If-Modified-Since' and 'If-Unmodified-Since'.
-	lastModified := objInfo.ModifiedTime
+	lastModified := objInfo.ModTime
 	if checkLastModified(w, r, lastModified) {
 		return
 	}
@@ -133,34 +124,22 @@ func (api objectStorageAPI) GetObjectHandler(w http.ResponseWriter, r *http.Requ
 		return
 	}
 
-	// Get the object.
-	startOffset := hrange.start
-	readCloser, err := api.ObjectAPI.GetObject(bucket, object, startOffset)
-	if err != nil {
-		errorIf(err.Trace(), "GetObject failed.", nil)
-		writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		return
-	}
-	defer readCloser.Close() // Close after this handler returns.
-
 	// Set standard object headers.
 	setObjectHeaders(w, objInfo, hrange)
 
 	// Set any additional requested response headers.
 	setGetRespHeaders(w, r.URL.Query())
 
-	if hrange.length > 0 {
-		if _, e := io.CopyN(w, readCloser, hrange.length); e != nil {
-			errorIf(probe.NewError(e), "Writing to client failed", nil)
-			// Do not send error response here, since client could have died.
-			return
-		}
-	} else {
-		if _, e := io.Copy(w, readCloser); e != nil {
-			errorIf(probe.NewError(e), "Writing to client failed", nil)
-			// Do not send error response here, since client could have died.
-			return
-		}
+	// Get the object.
+	startOffset := hrange.start
+	length := hrange.length
+	if length == 0 {
+		length = objInfo.Size - startOffset
+	}
+	if err := api.ObjectAPI.GetObject(bucket, object, startOffset, length, w); err != nil {
+		errorIf(err, "Writing to client failed.")
+		// Do not send error response here, client would have already died.
+		return
 	}
 }
 
@@ -261,7 +240,7 @@ func checkETag(w http.ResponseWriter, r *http.Request) bool {
 // HeadObjectHandler - HEAD Object
 // -----------
 // The HEAD operation retrieves metadata from an object without returning the object itself.
-func (api objectStorageAPI) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
 	var object, bucket string
 	vars := mux.Vars(r)
 	bucket = vars["bucket"]
@@ -287,24 +266,17 @@ func (api objectStorageAPI) HeadObjectHandler(w http.ResponseWriter, r *http.Req
 
 	objInfo, err := api.ObjectAPI.GetObjectInfo(bucket, object)
 	if err != nil {
-		errorIf(err.Trace(bucket, object), "GetObjectInfo failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, errAllowableObjectNotFound(bucket, r), r.URL.Path)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
+		errorIf(err, "Unable to fetch object info.")
+		apiErr := toAPIErrorCode(err)
+		if apiErr == ErrNoSuchKey {
+			apiErr = errAllowableObjectNotFound(bucket, r)
 		}
+		writeErrorResponse(w, r, apiErr, r.URL.Path)
 		return
 	}
 
 	// Verify 'If-Modified-Since' and 'If-Unmodified-Since'.
-	lastModified := objInfo.ModifiedTime
+	lastModified := objInfo.ModTime
 	if checkLastModified(w, r, lastModified) {
 		return
 	}
@@ -325,7 +297,7 @@ func (api objectStorageAPI) HeadObjectHandler(w http.ResponseWriter, r *http.Req
 // ----------
 // This implementation of the PUT operation adds an object to a bucket
 // while reading the object from another source.
-func (api objectStorageAPI) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 	object := vars["object"]
@@ -380,26 +352,15 @@ func (api objectStorageAPI) CopyObjectHandler(w http.ResponseWriter, r *http.Req
 
 	objInfo, err := api.ObjectAPI.GetObjectInfo(sourceBucket, sourceObject)
 	if err != nil {
-		errorIf(err.Trace(), "GetObjectInfo failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, objectSource)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, objectSource)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, ErrNoSuchKey, objectSource)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, objectSource)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, objectSource)
-		}
+		errorIf(err, "Unable to fetch object info.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), objectSource)
 		return
 	}
 	// Verify before writing.
 
 	// Verify x-amz-copy-source-if-modified-since and
 	// x-amz-copy-source-if-unmodified-since.
-	lastModified := objInfo.ModifiedTime
+	lastModified := objInfo.ModTime
 	if checkCopySourceLastModified(w, r, lastModified) {
 		return
 	}
@@ -416,69 +377,53 @@ func (api objectStorageAPI) CopyObjectHandler(w http.ResponseWriter, r *http.Req
 		return
 	}
 
-	var md5Bytes []byte
-	if objInfo.MD5Sum != "" {
-		var e error
-		md5Bytes, e = hex.DecodeString(objInfo.MD5Sum)
-		if e != nil {
-			errorIf(probe.NewError(e), "Decoding md5 failed.", nil)
-			writeErrorResponse(w, r, ErrInvalidDigest, r.URL.Path)
+	pipeReader, pipeWriter := io.Pipe()
+	go func() {
+		startOffset := int64(0) // Read the whole file.
+		// Get the object.
+		gErr := api.ObjectAPI.GetObject(sourceBucket, sourceObject, startOffset, objInfo.Size, pipeWriter)
+		if gErr != nil {
+			errorIf(gErr, "Unable to read an object.")
+			pipeWriter.CloseWithError(gErr)
 			return
 		}
-	}
+		pipeWriter.Close() // Close.
+	}()
 
-	startOffset := int64(0) // Read the whole file.
-	// Get the object.
-	readCloser, getErr := api.ObjectAPI.GetObject(sourceBucket, sourceObject, startOffset)
-	if getErr != nil {
-		errorIf(getErr.Trace(sourceBucket, sourceObject), "Reading "+objectSource+" failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, objectSource)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, ErrNoSuchKey, objectSource)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, objectSource)
-		}
-		return
-	}
 	// Size of object.
 	size := objInfo.Size
 
 	// Save metadata.
 	metadata := make(map[string]string)
-	metadata["md5Sum"] = hex.EncodeToString(md5Bytes)
+	// Save other metadata if available.
+	metadata["content-type"] = objInfo.ContentType
+	metadata["content-encoding"] = objInfo.ContentEncoding
+	// Do not set `md5sum` as CopyObject will not keep the
+	// same md5sum as the source.
 
 	// Create the object.
-	objInfo, err = api.ObjectAPI.PutObject(bucket, object, size, readCloser, metadata)
+	md5Sum, err := api.ObjectAPI.PutObject(bucket, object, size, pipeReader, metadata)
 	if err != nil {
-		errorIf(err.Trace(), "PutObject failed.", nil)
-		switch err.ToGoError().(type) {
-		case RootPathFull:
-			writeErrorResponse(w, r, ErrRootPathFull, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BadDigest:
-			writeErrorResponse(w, r, ErrBadDigest, r.URL.Path)
-		case IncompleteBody:
-			writeErrorResponse(w, r, ErrIncompleteBody, r.URL.Path)
-		case ObjectExistsAsPrefix:
-			writeErrorResponse(w, r, ErrObjectExistsAsPrefix, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		errorIf(err, "Unable to create an object.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
-	response := generateCopyObjectResponse(objInfo.MD5Sum, objInfo.ModifiedTime)
+
+	objInfo, err = api.ObjectAPI.GetObjectInfo(bucket, object)
+	if err != nil {
+		errorIf(err, "Unable to fetch object info.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
+		return
+	}
+
+	response := generateCopyObjectResponse(md5Sum, objInfo.ModTime)
 	encodedSuccessResponse := encodeResponse(response)
 	// write headers
 	setCommonHeaders(w)
 	// write success response.
 	writeSuccessResponse(w, encodedSuccessResponse)
 	// Explicitly close the reader, to avoid fd leaks.
-	readCloser.Close()
+	pipeReader.Close()
 }
 
 // checkCopySource implements x-amz-copy-source-if-modified-since and
@@ -576,7 +521,7 @@ func checkCopySourceETag(w http.ResponseWriter, r *http.Request) bool {
 // PutObjectHandler - PUT Object
 // ----------
 // This implementation of the PUT operation adds an object to a bucket.
-func (api objectStorageAPI) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
 	// If the matching failed, it means that the X-Amz-Copy-Source was
 	// wrong, fail right here.
 	if _, ok := r.Header["X-Amz-Copy-Source"]; ok {
@@ -590,7 +535,7 @@ func (api objectStorageAPI) PutObjectHandler(w http.ResponseWriter, r *http.Requ
 	// Get Content-Md5 sent by client and verify if valid
 	md5Bytes, err := checkValidMD5(r.Header.Get("Content-Md5"))
 	if err != nil {
-		errorIf(err.Trace(r.Header.Get("Content-Md5")), "Decoding md5 failed.", nil)
+		errorIf(err, "Unable to validate content-md5 format.")
 		writeErrorResponse(w, r, ErrInvalidDigest, r.URL.Path)
 		return
 	}
@@ -606,7 +551,23 @@ func (api objectStorageAPI) PutObjectHandler(w http.ResponseWriter, r *http.Requ
 		return
 	}
 
-	var objInfo ObjectInfo
+	// Save metadata.
+	metadata := make(map[string]string)
+	// Make sure we hex encode md5sum here.
+	metadata["md5Sum"] = hex.EncodeToString(md5Bytes)
+	// Save other metadata if available.
+	metadata["content-type"] = r.Header.Get("Content-Type")
+	metadata["content-encoding"] = r.Header.Get("Content-Encoding")
+	for key := range r.Header {
+		cKey := http.CanonicalHeaderKey(key)
+		if strings.HasPrefix(cKey, "x-amz-meta-") {
+			metadata[cKey] = r.Header.Get(cKey)
+		} else if strings.HasPrefix(key, "x-minio-meta-") {
+			metadata[cKey] = r.Header.Get(cKey)
+		}
+	}
+
+	var md5Sum string
 	switch getRequestAuthType(r) {
 	default:
 		// For all unknown auth types return error.
@@ -619,18 +580,24 @@ func (api objectStorageAPI) PutObjectHandler(w http.ResponseWriter, r *http.Requ
 			return
 		}
 		// Create anonymous object.
-		objInfo, err = api.ObjectAPI.PutObject(bucket, object, size, r.Body, nil)
+		md5Sum, err = api.ObjectAPI.PutObject(bucket, object, size, r.Body, metadata)
 	case authTypePresigned, authTypeSigned:
 		// Initialize a pipe for data pipe line.
 		reader, writer := io.Pipe()
-
+		var wg = &sync.WaitGroup{}
 		// Start writing in a routine.
+		wg.Add(1)
 		go func() {
-			shaWriter := fastSha256.New()
+			defer wg.Done()
+			shaWriter := sha256.New()
 			multiWriter := io.MultiWriter(shaWriter, writer)
-			if _, e := io.CopyN(multiWriter, r.Body, size); e != nil {
-				errorIf(probe.NewError(e), "Unable to read HTTP body.", nil)
-				writer.CloseWithError(e)
+			if _, wErr := io.CopyN(multiWriter, r.Body, size); wErr != nil {
+				// Pipe closed.
+				if wErr == io.ErrClosedPipe {
+					return
+				}
+				errorIf(wErr, "Unable to read from HTTP body.")
+				writer.CloseWithError(wErr)
 				return
 			}
 			shaPayload := shaWriter.Sum(nil)
@@ -641,61 +608,41 @@ func (api objectStorageAPI) PutObjectHandler(w http.ResponseWriter, r *http.Requ
 			} else if isRequestPresignedSignatureV4(r) {
 				s3Error = doesPresignedSignatureMatch(hex.EncodeToString(shaPayload), r, validateRegion)
 			}
+			var sErr error
 			if s3Error != ErrNone {
 				if s3Error == ErrSignatureDoesNotMatch {
-					writer.CloseWithError(errSignatureMismatch)
-					return
+					sErr = errSignatureMismatch
+				} else {
+					sErr = fmt.Errorf("%v", getAPIError(s3Error))
 				}
-				writer.CloseWithError(fmt.Errorf("%v", getAPIError(s3Error)))
+				writer.CloseWithError(sErr)
 				return
 			}
-			// Close the writer.
 			writer.Close()
 		}()
 
-		// Save metadata.
-		metadata := make(map[string]string)
-		// Make sure we hex encode here.
-		metadata["md5"] = hex.EncodeToString(md5Bytes)
 		// Create object.
-		objInfo, err = api.ObjectAPI.PutObject(bucket, object, size, reader, metadata)
+		md5Sum, err = api.ObjectAPI.PutObject(bucket, object, size, reader, metadata)
+		// Close the pipe.
+		reader.Close()
+		// Wait for all the routines to finish.
+		wg.Wait()
 	}
 	if err != nil {
-		errorIf(err.Trace(), "PutObject failed.", nil)
-		e := err.ToGoError()
-		// Verify if the underlying error is signature mismatch.
-		if e == errSignatureMismatch {
-			writeErrorResponse(w, r, ErrSignatureDoesNotMatch, r.URL.Path)
-			return
-		}
-		switch e.(type) {
-		case RootPathFull:
-			writeErrorResponse(w, r, ErrRootPathFull, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BadDigest:
-			writeErrorResponse(w, r, ErrBadDigest, r.URL.Path)
-		case IncompleteBody:
-			writeErrorResponse(w, r, ErrIncompleteBody, r.URL.Path)
-		case ObjectExistsAsPrefix:
-			writeErrorResponse(w, r, ErrObjectExistsAsPrefix, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		errorIf(err, "Unable to create an object.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
-	if objInfo.MD5Sum != "" {
-		w.Header().Set("ETag", "\""+objInfo.MD5Sum+"\"")
+	if md5Sum != "" {
+		w.Header().Set("ETag", "\""+md5Sum+"\"")
 	}
 	writeSuccessResponse(w, nil)
 }
 
-/// Multipart objectStorageAPI
+/// Multipart objectAPIHandlers
 
 // NewMultipartUploadHandler - New multipart upload
-func (api objectStorageAPI) NewMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) NewMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
 	var object, bucket string
 	vars := mux.Vars(r)
 	bucket = vars["bucket"]
@@ -719,23 +666,24 @@ func (api objectStorageAPI) NewMultipartUploadHandler(w http.ResponseWriter, r *
 		}
 	}
 
-	uploadID, err := api.ObjectAPI.NewMultipartUpload(bucket, object)
-	if err != nil {
-		errorIf(err.Trace(), "NewMultipartUpload failed.", nil)
-		switch err.ToGoError().(type) {
-		case RootPathFull:
-			writeErrorResponse(w, r, ErrRootPathFull, r.URL.Path)
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
+	// Save metadata.
+	metadata := make(map[string]string)
+	// Save other metadata if available.
+	metadata["content-type"] = r.Header.Get("Content-Type")
+	metadata["content-encoding"] = r.Header.Get("Content-Encoding")
+	for key := range r.Header {
+		cKey := http.CanonicalHeaderKey(key)
+		if strings.HasPrefix(cKey, "x-amz-meta-") {
+			metadata[cKey] = r.Header.Get(cKey)
+		} else if strings.HasPrefix(key, "x-minio-meta-") {
+			metadata[cKey] = r.Header.Get(cKey)
 		}
+	}
+
+	uploadID, err := api.ObjectAPI.NewMultipartUpload(bucket, object, metadata)
+	if err != nil {
+		errorIf(err, "Unable to initiate new multipart upload id.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 
@@ -748,7 +696,7 @@ func (api objectStorageAPI) NewMultipartUploadHandler(w http.ResponseWriter, r *
 }
 
 // PutObjectPartHandler - Upload part
-func (api objectStorageAPI) PutObjectPartHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) PutObjectPartHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 	object := vars["object"]
@@ -756,7 +704,6 @@ func (api objectStorageAPI) PutObjectPartHandler(w http.ResponseWriter, r *http.
 	// get Content-Md5 sent by client and verify if valid
 	md5Bytes, err := checkValidMD5(r.Header.Get("Content-Md5"))
 	if err != nil {
-		errorIf(err.Trace(r.Header.Get("Content-Md5")), "Decoding md5 failed.", nil)
 		writeErrorResponse(w, r, ErrInvalidDigest, r.URL.Path)
 		return
 	}
@@ -777,12 +724,18 @@ func (api objectStorageAPI) PutObjectPartHandler(w http.ResponseWriter, r *http.
 	uploadID := r.URL.Query().Get("uploadId")
 	partIDString := r.URL.Query().Get("partNumber")
 
-	partID, e := strconv.Atoi(partIDString)
-	if e != nil {
+	partID, err := strconv.Atoi(partIDString)
+	if err != nil {
 		writeErrorResponse(w, r, ErrInvalidPart, r.URL.Path)
 		return
 	}
 
+	// check partID with maximum part ID for multipart objects
+	if isMaxPartID(partID) {
+		writeErrorResponse(w, r, ErrInvalidMaxParts, r.URL.Path)
+		return
+	}
+
 	var partMD5 string
 	switch getRequestAuthType(r) {
 	default:
@@ -797,22 +750,29 @@ func (api objectStorageAPI) PutObjectPartHandler(w http.ResponseWriter, r *http.
 		}
 		// No need to verify signature, anonymous request access is
 		// already allowed.
-		partMD5, err = api.ObjectAPI.PutObjectPart(bucket, object, uploadID, partID, size, r.Body, hex.EncodeToString(md5Bytes))
+		hexMD5 := hex.EncodeToString(md5Bytes)
+		partMD5, err = api.ObjectAPI.PutObjectPart(bucket, object, uploadID, partID, size, r.Body, hexMD5)
 	case authTypePresigned, authTypeSigned:
-		validateRegion := true // Validate region.
 		// Initialize a pipe for data pipe line.
 		reader, writer := io.Pipe()
-
+		var wg = &sync.WaitGroup{}
 		// Start writing in a routine.
+		wg.Add(1)
 		go func() {
-			shaWriter := fastSha256.New()
+			defer wg.Done()
+			shaWriter := sha256.New()
 			multiWriter := io.MultiWriter(shaWriter, writer)
-			if _, e := io.CopyN(multiWriter, r.Body, size); e != nil {
-				errorIf(probe.NewError(e), "Unable to read HTTP body.", nil)
-				writer.CloseWithError(e)
+			if _, wErr := io.CopyN(multiWriter, r.Body, size); wErr != nil {
+				// Pipe closed, just ignore it.
+				if wErr == io.ErrClosedPipe {
+					return
+				}
+				errorIf(wErr, "Unable to read from HTTP request body.")
+				writer.CloseWithError(wErr)
 				return
 			}
 			shaPayload := shaWriter.Sum(nil)
+			validateRegion := true // Validate region.
 			var s3Error APIErrorCode
 			if isRequestSignatureV4(r) {
 				s3Error = doesSignatureMatch(hex.EncodeToString(shaPayload), r, validateRegion)
@@ -821,37 +781,27 @@ func (api objectStorageAPI) PutObjectPartHandler(w http.ResponseWriter, r *http.
 			}
 			if s3Error != ErrNone {
 				if s3Error == ErrSignatureDoesNotMatch {
-					writer.CloseWithError(errSignatureMismatch)
-					return
+					err = errSignatureMismatch
+				} else {
+					err = fmt.Errorf("%v", getAPIError(s3Error))
 				}
-				writer.CloseWithError(fmt.Errorf("%v", getAPIError(s3Error)))
+				writer.CloseWithError(err)
 				return
 			}
 			// Close the writer.
 			writer.Close()
 		}()
-		partMD5, err = api.ObjectAPI.PutObjectPart(bucket, object, uploadID, partID, size, reader, hex.EncodeToString(md5Bytes))
+		md5SumHex := hex.EncodeToString(md5Bytes)
+		partMD5, err = api.ObjectAPI.PutObjectPart(bucket, object, uploadID, partID, size, reader, md5SumHex)
+		// Close the pipe.
+		reader.Close()
+		// Wait for all the routines to finish.
+		wg.Wait()
 	}
 	if err != nil {
-		errorIf(err.Trace(), "PutObjectPart failed.", nil)
-		e := err.ToGoError()
+		errorIf(err, "Unable to create object part.")
 		// Verify if the underlying error is signature mismatch.
-		if e == errSignatureMismatch {
-			writeErrorResponse(w, r, ErrSignatureDoesNotMatch, r.URL.Path)
-			return
-		}
-		switch e.(type) {
-		case RootPathFull:
-			writeErrorResponse(w, r, ErrRootPathFull, r.URL.Path)
-		case InvalidUploadID:
-			writeErrorResponse(w, r, ErrNoSuchUpload, r.URL.Path)
-		case BadDigest:
-			writeErrorResponse(w, r, ErrBadDigest, r.URL.Path)
-		case IncompleteBody:
-			writeErrorResponse(w, r, ErrIncompleteBody, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 	if partMD5 != "" {
@@ -861,7 +811,7 @@ func (api objectStorageAPI) PutObjectPartHandler(w http.ResponseWriter, r *http.
 }
 
 // AbortMultipartUploadHandler - Abort multipart upload
-func (api objectStorageAPI) AbortMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) AbortMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 	object := vars["object"]
@@ -885,30 +835,16 @@ func (api objectStorageAPI) AbortMultipartUploadHandler(w http.ResponseWriter, r
 	}
 
 	uploadID, _, _, _ := getObjectResources(r.URL.Query())
-	err := api.ObjectAPI.AbortMultipartUpload(bucket, object, uploadID)
-	if err != nil {
-		errorIf(err.Trace(), "AbortMutlipartUpload failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case InvalidUploadID:
-			writeErrorResponse(w, r, ErrNoSuchUpload, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+	if err := api.ObjectAPI.AbortMultipartUpload(bucket, object, uploadID); err != nil {
+		errorIf(err, "Unable to abort multipart upload.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 	writeSuccessNoContent(w)
 }
 
 // ListObjectPartsHandler - List object parts
-func (api objectStorageAPI) ListObjectPartsHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) ListObjectPartsHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 	object := vars["object"]
@@ -940,27 +876,10 @@ func (api objectStorageAPI) ListObjectPartsHandler(w http.ResponseWriter, r *htt
 		writeErrorResponse(w, r, ErrInvalidMaxParts, r.URL.Path)
 		return
 	}
-	if maxParts == 0 {
-		maxParts = maxPartsList
-	}
-
 	listPartsInfo, err := api.ObjectAPI.ListObjectParts(bucket, object, uploadID, partNumberMarker, maxParts)
 	if err != nil {
-		errorIf(err.Trace(), "ListObjectParts failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case InvalidUploadID:
-			writeErrorResponse(w, r, ErrNoSuchUpload, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		errorIf(err, "Unable to list uploaded parts.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 	response := generateListPartsResponse(listPartsInfo)
@@ -972,7 +891,7 @@ func (api objectStorageAPI) ListObjectPartsHandler(w http.ResponseWriter, r *htt
 }
 
 // CompleteMultipartUploadHandler - Complete multipart upload
-func (api objectStorageAPI) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 	object := vars["object"]
@@ -980,8 +899,8 @@ func (api objectStorageAPI) CompleteMultipartUploadHandler(w http.ResponseWriter
 	// Get upload id.
 	uploadID, _, _, _ := getObjectResources(r.URL.Query())
 
-	var objInfo ObjectInfo
-	var err *probe.Error
+	var md5Sum string
+	var err error
 	switch getRequestAuthType(r) {
 	default:
 		// For all unknown auth types return error.
@@ -999,14 +918,19 @@ func (api objectStorageAPI) CompleteMultipartUploadHandler(w http.ResponseWriter
 			return
 		}
 	}
-	completeMultipartBytes, e := ioutil.ReadAll(r.Body)
-	if e != nil {
-		errorIf(probe.NewError(e), "CompleteMultipartUpload failed.", nil)
+	completeMultipartBytes, err := ioutil.ReadAll(r.Body)
+	if err != nil {
+		errorIf(err, "Unable to complete multipart upload.")
 		writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
 		return
 	}
 	complMultipartUpload := &completeMultipartUpload{}
-	if e = xml.Unmarshal(completeMultipartBytes, complMultipartUpload); e != nil {
+	if err = xml.Unmarshal(completeMultipartBytes, complMultipartUpload); err != nil {
+		errorIf(err, "Unable to parse complete multipart upload XML.")
+		writeErrorResponse(w, r, ErrMalformedXML, r.URL.Path)
+		return
+	}
+	if len(complMultipartUpload.Parts) == 0 {
 		writeErrorResponse(w, r, ErrMalformedXML, r.URL.Path)
 		return
 	}
@@ -1015,38 +939,23 @@ func (api objectStorageAPI) CompleteMultipartUploadHandler(w http.ResponseWriter
 		return
 	}
 	// Complete parts.
-	completeParts := complMultipartUpload.Parts
-
+	var completeParts []completePart
+	for _, part := range complMultipartUpload.Parts {
+		part.ETag = strings.TrimPrefix(part.ETag, "\"")
+		part.ETag = strings.TrimSuffix(part.ETag, "\"")
+		completeParts = append(completeParts, part)
+	}
 	// Complete multipart upload.
-	objInfo, err = api.ObjectAPI.CompleteMultipartUpload(bucket, object, uploadID, completeParts)
+	md5Sum, err = api.ObjectAPI.CompleteMultipartUpload(bucket, object, uploadID, completeParts)
 	if err != nil {
-		errorIf(err.Trace(), "CompleteMultipartUpload failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case InvalidUploadID:
-			writeErrorResponse(w, r, ErrNoSuchUpload, r.URL.Path)
-		case InvalidPart:
-			writeErrorResponse(w, r, ErrInvalidPart, r.URL.Path)
-		case InvalidPartOrder:
-			writeErrorResponse(w, r, ErrInvalidPartOrder, r.URL.Path)
-		case IncompleteBody:
-			writeErrorResponse(w, r, ErrIncompleteBody, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
+		errorIf(err, "Unable to complete multipart upload.")
+		writeErrorResponse(w, r, toAPIErrorCode(err), r.URL.Path)
 		return
 	}
 	// Get object location.
 	location := getLocation(r)
 	// Generate complete multipart response.
-	response := generateCompleteMultpartUploadResponse(bucket, object, location, objInfo.MD5Sum)
+	response := generateCompleteMultpartUploadResponse(bucket, object, location, md5Sum)
 	encodedSuccessResponse := encodeResponse(response)
 	// Write headers.
 	setCommonHeaders(w)
@@ -1054,10 +963,10 @@ func (api objectStorageAPI) CompleteMultipartUploadHandler(w http.ResponseWriter
 	writeSuccessResponse(w, encodedSuccessResponse)
 }
 
-/// Delete objectStorageAPI
+/// Delete objectAPIHandlers
 
 // DeleteObjectHandler - delete an object
-func (api objectStorageAPI) DeleteObjectHandler(w http.ResponseWriter, r *http.Request) {
+func (api objectAPIHandlers) DeleteObjectHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
 	object := vars["object"]
@@ -1079,22 +988,9 @@ func (api objectStorageAPI) DeleteObjectHandler(w http.ResponseWriter, r *http.R
 			return
 		}
 	}
-	err := api.ObjectAPI.DeleteObject(bucket, object)
-	if err != nil {
-		errorIf(err.Trace(), "DeleteObject failed.", nil)
-		switch err.ToGoError().(type) {
-		case BucketNameInvalid:
-			writeErrorResponse(w, r, ErrInvalidBucketName, r.URL.Path)
-		case BucketNotFound:
-			writeErrorResponse(w, r, ErrNoSuchBucket, r.URL.Path)
-		case ObjectNotFound:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		case ObjectNameInvalid:
-			writeErrorResponse(w, r, ErrNoSuchKey, r.URL.Path)
-		default:
-			writeErrorResponse(w, r, ErrInternalError, r.URL.Path)
-		}
-		return
-	}
+	/// http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETE.html
+	/// Ignore delete object errors, since we are suppposed to reply
+	/// only 204.
+	api.ObjectAPI.DeleteObject(bucket, object)
 	writeSuccessNoContent(w)
 }

object-interface.go

@@ -1 +1,46 @@
+/*
+ * Minio Cloud Storage, (C) 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 package main
+
+import "io"
+
+// ObjectLayer implements primitives for object API layer.
+type ObjectLayer interface {
+	// Storage operations.
+	StorageInfo() StorageInfo
+
+	// Bucket operations.
+	MakeBucket(bucket string) error
+	GetBucketInfo(bucket string) (bucketInfo BucketInfo, err error)
+	ListBuckets() (buckets []BucketInfo, err error)
+	DeleteBucket(bucket string) error
+	ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (result ListObjectsInfo, err error)
+
+	// Object operations.
+	GetObject(bucket, object string, startOffset int64, length int64, writer io.Writer) (err error)
+	GetObjectInfo(bucket, object string) (objInfo ObjectInfo, err error)
+	PutObject(bucket, object string, size int64, data io.Reader, metadata map[string]string) (md5 string, err error)
+	DeleteObject(bucket, object string) error
+
+	// Multipart operations.
+	ListMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartsInfo, err error)
+	NewMultipartUpload(bucket, object string, metadata map[string]string) (uploadID string, err error)
+	PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (md5 string, err error)
+	ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (result ListPartsInfo, err error)
+	AbortMultipartUpload(bucket, object, uploadID string) error
+	CompleteMultipartUpload(bucket, object, uploadID string, uploadedParts []completePart) (md5 string, err error)
+}

object-utils.go

@@ -0,0 +1,159 @@
+/*
+ * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"crypto/md5"
+	"encoding/hex"
+	"fmt"
+	"path"
+	"regexp"
+	"strings"
+	"unicode/utf8"
+
+	"github.com/skyrings/skyring-common/tools/uuid"
+)
+
+const (
+	// Minio meta bucket.
+	minioMetaBucket = ".minio"
+	// Multipart meta prefix.
+	mpartMetaPrefix = "multipart"
+	// Tmp meta prefix.
+	tmpMetaPrefix = "tmp"
+)
+
+// validBucket regexp.
+var validBucket = regexp.MustCompile(`^[a-z0-9][a-z0-9\.\-]{1,61}[a-z0-9]$`)
+
+// IsValidBucketName verifies a bucket name in accordance with Amazon's
+// requirements. It must be 3-63 characters long, can contain dashes
+// and periods, but must begin and end with a lowercase letter or a number.
+// See: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
+func IsValidBucketName(bucket string) bool {
+	if len(bucket) < 3 || len(bucket) > 63 {
+		return false
+	}
+	if bucket[0] == '.' || bucket[len(bucket)-1] == '.' {
+		return false
+	}
+	return validBucket.MatchString(bucket)
+}
+
+// IsValidObjectName verifies an object name in accordance with Amazon's
+// requirements. It cannot exceed 1024 characters and must be a valid UTF8
+// string.
+//
+// See:
+// http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
+//
+// You should avoid the following characters in a key name because of
+// significant special handling for consistency across all
+// applications.
+//
+// Rejects strings with following characters.
+//
+// - Backslash ("\")
+// - Caret ("^")
+// - Grave accent / back tick ("`")
+// - Vertical bar / pipe ("|")
+//
+// Minio does not support object names with trailing "/".
+func IsValidObjectName(object string) bool {
+	if len(object) == 0 {
+		return false
+	}
+	if strings.HasSuffix(object, slashSeparator) {
+		return false
+	}
+	if strings.HasPrefix(object, slashSeparator) {
+		return false
+	}
+	return IsValidObjectPrefix(object)
+}
+
+// IsValidObjectPrefix verifies whether the prefix is a valid object name.
+// Its valid to have a empty prefix.
+func IsValidObjectPrefix(object string) bool {
+	if len(object) > 1024 {
+		return false
+	}
+	if !utf8.ValidString(object) {
+		return false
+	}
+	// Reject unsupported characters in object name.
+	if strings.ContainsAny(object, "`^*|\\\"") {
+		return false
+	}
+	return true
+}
+
+// Slash separator.
+const slashSeparator = "/"
+
+// retainSlash - retains slash from a path.
+func retainSlash(s string) string {
+	return strings.TrimSuffix(s, slashSeparator) + slashSeparator
+}
+
+// pathJoin - like path.Join() but retains trailing "/" of the last element
+func pathJoin(elem ...string) string {
+	trailingSlash := ""
+	if len(elem) > 0 {
+		if strings.HasSuffix(elem[len(elem)-1], slashSeparator) {
+			trailingSlash = "/"
+		}
+	}
+	return path.Join(elem...) + trailingSlash
+}
+
+// getUUID() - get a unique uuid.
+func getUUID() (uuidStr string) {
+	for {
+		uuid, err := uuid.New()
+		if err != nil {
+			errorIf(err, "Unable to initialize uuid")
+			continue
+		}
+		uuidStr = uuid.String()
+		break
+	}
+	return uuidStr
+}
+
+// Create an s3 compatible MD5sum for complete multipart transaction.
+func completeMultipartMD5(parts ...completePart) (string, error) {
+	var finalMD5Bytes []byte
+	for _, part := range parts {
+		md5Bytes, err := hex.DecodeString(part.ETag)
+		if err != nil {
+			return "", err
+		}
+		finalMD5Bytes = append(finalMD5Bytes, md5Bytes...)
+	}
+	md5Hasher := md5.New()
+	md5Hasher.Write(finalMD5Bytes)
+	s3MD5 := fmt.Sprintf("%s-%d", hex.EncodeToString(md5Hasher.Sum(nil)), len(parts))
+	return s3MD5, nil
+}
+
+// byBucketName is a collection satisfying sort.Interface.
+type byBucketName []BucketInfo
+
+func (d byBucketName) Len() int           { return len(d) }
+func (d byBucketName) Swap(i, j int)      { d[i], d[j] = d[j], d[i] }
+func (d byBucketName) Less(i, j int) bool { return d[i].Name < d[j].Name }

object-utils_test.go

@@ -20,24 +20,14 @@ import (
 	"testing"
 )
 
-//Validating bucket name.
-func ensureBucketName(t *testing.T, name string, testNum int, pass bool) {
-	isValidBucketName := IsValidBucketName(name)
-	if pass && !isValidBucketName {
-		t.Errorf("Test case %d: Expected \"%s\" to be a valid bucket name", testNum, name)
-	}
-	if !pass && isValidBucketName {
-		t.Errorf("Test case %d: Expected bucket name \"%s\" to be invalid", testNum, name)
-	}
-}
-
+// Tests validate bucket name.
 func TestIsValidBucketName(t *testing.T) {
 	testCases := []struct {
 		bucketName string
 		shouldPass bool
 	}{
-		//cases which should pass the test
-		//passing in valid bucket names
+		// cases which should pass the test.
+		// passing in valid bucket names.
 		{"lol", true},
 		{"1-this-is-valid", true},
 		{"1-this-too-is-valid-1", true},
@@ -49,8 +39,8 @@ func TestIsValidBucketName(t *testing.T) {
 		{"testbucket", true},
 		{"1bucket", true},
 		{"bucket1", true},
-		//cases for which test should fail
-		//passing invalid bucket names
+		// cases for which test should fail.
+		// passing invalid bucket names.
 		{"------", false},
 		{"$this-is-not-valid-too", false},
 		{"contains-$-dollar", false},
@@ -73,42 +63,45 @@ func TestIsValidBucketName(t *testing.T) {
 	}
 
 	for i, testCase := range testCases {
-		ensureBucketName(t, testCase.bucketName, i+1, testCase.shouldPass)
+		isValidBucketName := IsValidBucketName(testCase.bucketName)
+		if testCase.shouldPass && !isValidBucketName {
+			t.Errorf("Test case %d: Expected \"%s\" to be a valid bucket name", i+1, testCase.bucketName)
+		}
+		if !testCase.shouldPass && isValidBucketName {
+			t.Errorf("Test case %d: Expected bucket name \"%s\" to be invalid", i+1, testCase.bucketName)
+		}
 	}
 }
 
-//Test for validating object name.
-func ensureObjectName(t *testing.T, name string, testNum int, pass bool) {
-	isValidObjectName := IsValidObjectName(name)
-	if pass && !isValidObjectName {
-		t.Errorf("Test case %d: Expected \"%s\" to be a valid object name", testNum, name)
-	}
-	if !pass && isValidObjectName {
-		t.Errorf("Test case %d: Expected object name \"%s\" to be invalid", testNum, name)
-	}
-
-}
-
+// Tests for validate object name.
 func TestIsValidObjectName(t *testing.T) {
 	testCases := []struct {
 		objectName string
 		shouldPass bool
 	}{
-		//cases which should pass the test
-		//passing in valid object name
+		// cases which should pass the test.
+		// passing in valid object name.
 		{"object", true},
 		{"The Shining Script <v1>.pdf", true},
 		{"Cost Benefit Analysis (2009-2010).pptx", true},
 		{"117Gn8rfHL2ACARPAhaFd0AGzic9pUbIA/5OCn5A", true},
 		{"SHØRT", true},
 		{"There are far too many object names, and far too few bucket names!", true},
-		//cases for which test should fail
-		//passing invalid object names
+		// cases for which test should fail.
+		// passing invalid object names.
 		{"", false},
+		{"a/b/c/", false},
+		{"/a/b/c", false},
 		{string([]byte{0xff, 0xfe, 0xfd}), false},
 	}
 
 	for i, testCase := range testCases {
-		ensureObjectName(t, testCase.objectName, i+1, testCase.shouldPass)
+		isValidObjectName := IsValidObjectName(testCase.objectName)
+		if testCase.shouldPass && !isValidObjectName {
+			t.Errorf("Test case %d: Expected \"%s\" to be a valid object name", i+1, testCase.objectName)
+		}
+		if !testCase.shouldPass && isValidObjectName {
+			t.Errorf("Test case %d: Expected object name \"%s\" to be invalid", i+1, testCase.objectName)
+		}
 	}
 }

object_api_suite_test.go

@@ -0,0 +1,432 @@
+/*
+ * Minio Cloud Storage, (C) 2015, 2016 Minio, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package main
+
+import (
+	"bytes"
+	"crypto/md5"
+	"encoding/hex"
+	"math/rand"
+	"strconv"
+
+	"gopkg.in/check.v1"
+)
+
+// APITestSuite - collection of API tests.
+func APITestSuite(c *check.C, create func() ObjectLayer) {
+	testMakeBucket(c, create)
+	testMultipleObjectCreation(c, create)
+	testPaging(c, create)
+	testObjectOverwriteWorks(c, create)
+	testNonExistantBucketOperations(c, create)
+	testBucketRecreateFails(c, create)
+	testPutObjectInSubdir(c, create)
+	testListBuckets(c, create)
+	testListBucketsOrder(c, create)
+	testListObjectsTestsForNonExistantBucket(c, create)
+	testNonExistantObjectInBucket(c, create)
+	testGetDirectoryReturnsObjectNotFound(c, create)
+	testDefaultContentType(c, create)
+	testMultipartObjectCreation(c, create)
+	testMultipartObjectAbort(c, create)
+}
+
+// Tests validate bucket creation.
+func testMakeBucket(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket-unknown")
+	c.Assert(err, check.IsNil)
+
+}
+
+// Tests validate creation of part files during Multipart operation.
+func testMultipartObjectCreation(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+	uploadID, err := obj.NewMultipartUpload("bucket", "key", nil)
+	c.Assert(err, check.IsNil)
+	// Create a byte array of 5MB.
+	data := bytes.Repeat([]byte("0123456789abcdef"), 5*1024*1024/16)
+	completedParts := completeMultipartUpload{}
+	for i := 1; i <= 10; i++ {
+		hasher := md5.New()
+		hasher.Write(data)
+		expectedMD5Sumhex := hex.EncodeToString(hasher.Sum(nil))
+
+		var calculatedMD5sum string
+		calculatedMD5sum, err = obj.PutObjectPart("bucket", "key", uploadID, i, int64(len(data)), bytes.NewBuffer(data), expectedMD5Sumhex)
+		c.Assert(err, check.IsNil)
+		c.Assert(calculatedMD5sum, check.Equals, expectedMD5Sumhex)
+		completedParts.Parts = append(completedParts.Parts, completePart{PartNumber: i, ETag: calculatedMD5sum})
+	}
+	md5Sum, err := obj.CompleteMultipartUpload("bucket", "key", uploadID, completedParts.Parts)
+	c.Assert(err, check.IsNil)
+	c.Assert(md5Sum, check.Equals, "7d364cb728ce42a74a96d22949beefb2-10")
+}
+
+// Tests validate abortion of Multipart operation.
+func testMultipartObjectAbort(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+	uploadID, err := obj.NewMultipartUpload("bucket", "key", nil)
+	c.Assert(err, check.IsNil)
+
+	parts := make(map[int]string)
+	metadata := make(map[string]string)
+	for i := 1; i <= 10; i++ {
+		randomPerm := rand.Perm(10)
+		randomString := ""
+		for _, num := range randomPerm {
+			randomString = randomString + strconv.Itoa(num)
+		}
+
+		hasher := md5.New()
+		hasher.Write([]byte(randomString))
+		expectedMD5Sumhex := hex.EncodeToString(hasher.Sum(nil))
+
+		metadata["md5"] = expectedMD5Sumhex
+		var calculatedMD5sum string
+		calculatedMD5sum, err = obj.PutObjectPart("bucket", "key", uploadID, i, int64(len(randomString)), bytes.NewBufferString(randomString), expectedMD5Sumhex)
+		c.Assert(err, check.IsNil)
+		c.Assert(calculatedMD5sum, check.Equals, expectedMD5Sumhex)
+		parts[i] = expectedMD5Sumhex
+	}
+	err = obj.AbortMultipartUpload("bucket", "key", uploadID)
+	c.Assert(err, check.IsNil)
+}
+
+// Tests validate object creation.
+func testMultipleObjectCreation(c *check.C, create func() ObjectLayer) {
+	objects := make(map[string][]byte)
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+	for i := 0; i < 10; i++ {
+		randomPerm := rand.Perm(100)
+		randomString := ""
+		for _, num := range randomPerm {
+			randomString = randomString + strconv.Itoa(num)
+		}
+
+		hasher := md5.New()
+		hasher.Write([]byte(randomString))
+		expectedMD5Sumhex := hex.EncodeToString(hasher.Sum(nil))
+
+		key := "obj" + strconv.Itoa(i)
+		objects[key] = []byte(randomString)
+		metadata := make(map[string]string)
+		metadata["md5Sum"] = expectedMD5Sumhex
+		var md5Sum string
+		md5Sum, err = obj.PutObject("bucket", key, int64(len(randomString)), bytes.NewBufferString(randomString), metadata)
+		c.Assert(err, check.IsNil)
+		c.Assert(md5Sum, check.Equals, expectedMD5Sumhex)
+	}
+
+	for key, value := range objects {
+		var byteBuffer bytes.Buffer
+		err = obj.GetObject("bucket", key, 0, int64(len(value)), &byteBuffer)
+		c.Assert(err, check.IsNil)
+		c.Assert(byteBuffer.Bytes(), check.DeepEquals, value)
+
+		objInfo, err := obj.GetObjectInfo("bucket", key)
+		c.Assert(err, check.IsNil)
+		c.Assert(objInfo.Size, check.Equals, int64(len(value)))
+	}
+}
+
+// Tests validate creation of objects and the order of listing using various filters for ListObjects operation.
+func testPaging(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	obj.MakeBucket("bucket")
+	result, err := obj.ListObjects("bucket", "", "", "", 0)
+	c.Assert(err, check.IsNil)
+	c.Assert(len(result.Objects), check.Equals, 0)
+	c.Assert(result.IsTruncated, check.Equals, false)
+	// check before paging occurs.
+	for i := 0; i < 5; i++ {
+		key := "obj" + strconv.Itoa(i)
+		_, err = obj.PutObject("bucket", key, int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+		c.Assert(err, check.IsNil)
+
+		result, err = obj.ListObjects("bucket", "", "", "", 5)
+		c.Assert(err, check.IsNil)
+		c.Assert(len(result.Objects), check.Equals, i+1)
+		c.Assert(result.IsTruncated, check.Equals, false)
+	}
+	// check after paging occurs pages work.
+	for i := 6; i <= 10; i++ {
+		key := "obj" + strconv.Itoa(i)
+		_, err = obj.PutObject("bucket", key, int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+		c.Assert(err, check.IsNil)
+		result, err = obj.ListObjects("bucket", "obj", "", "", 5)
+		c.Assert(err, check.IsNil)
+		c.Assert(len(result.Objects), check.Equals, 5)
+		c.Assert(result.IsTruncated, check.Equals, true)
+	}
+	// check paging with prefix at end returns less objects.
+	{
+		_, err = obj.PutObject("bucket", "newPrefix", int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+		c.Assert(err, check.IsNil)
+		_, err = obj.PutObject("bucket", "newPrefix2", int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+		c.Assert(err, check.IsNil)
+		result, err = obj.ListObjects("bucket", "new", "", "", 5)
+		c.Assert(err, check.IsNil)
+		c.Assert(len(result.Objects), check.Equals, 2)
+	}
+
+	// check ordering of pages.
+	{
+		result, err = obj.ListObjects("bucket", "", "", "", 1000)
+		c.Assert(err, check.IsNil)
+		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix")
+		c.Assert(result.Objects[1].Name, check.Equals, "newPrefix2")
+		c.Assert(result.Objects[2].Name, check.Equals, "obj0")
+		c.Assert(result.Objects[3].Name, check.Equals, "obj1")
+		c.Assert(result.Objects[4].Name, check.Equals, "obj10")
+	}
+
+	// check delimited results with delimiter and prefix.
+	{
+		_, err = obj.PutObject("bucket", "this/is/delimited", int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+		c.Assert(err, check.IsNil)
+		_, err = obj.PutObject("bucket", "this/is/also/a/delimited/file", int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+		c.Assert(err, check.IsNil)
+		result, err = obj.ListObjects("bucket", "this/is/", "", "/", 10)
+		c.Assert(err, check.IsNil)
+		c.Assert(len(result.Objects), check.Equals, 1)
+		c.Assert(result.Prefixes[0], check.Equals, "this/is/also/")
+	}
+
+	// check delimited results with delimiter without prefix.
+	{
+		result, err = obj.ListObjects("bucket", "", "", "/", 1000)
+		c.Assert(err, check.IsNil)
+		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix")
+		c.Assert(result.Objects[1].Name, check.Equals, "newPrefix2")
+		c.Assert(result.Objects[2].Name, check.Equals, "obj0")
+		c.Assert(result.Objects[3].Name, check.Equals, "obj1")
+		c.Assert(result.Objects[4].Name, check.Equals, "obj10")
+		c.Assert(result.Prefixes[0], check.Equals, "this/")
+	}
+
+	// check results with Marker.
+	{
+
+		result, err = obj.ListObjects("bucket", "", "newPrefix", "", 3)
+		c.Assert(err, check.IsNil)
+		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix2")
+		c.Assert(result.Objects[1].Name, check.Equals, "obj0")
+		c.Assert(result.Objects[2].Name, check.Equals, "obj1")
+
+	}
+	// check ordering of results with prefix.
+	{
+		result, err = obj.ListObjects("bucket", "obj", "", "", 1000)
+		c.Assert(err, check.IsNil)
+		c.Assert(result.Objects[0].Name, check.Equals, "obj0")
+		c.Assert(result.Objects[1].Name, check.Equals, "obj1")
+		c.Assert(result.Objects[2].Name, check.Equals, "obj10")
+		c.Assert(result.Objects[3].Name, check.Equals, "obj2")
+		c.Assert(result.Objects[4].Name, check.Equals, "obj3")
+	}
+	// check ordering of results with prefix and no paging.
+	{
+		result, err = obj.ListObjects("bucket", "new", "", "", 5)
+		c.Assert(err, check.IsNil)
+		c.Assert(result.Objects[0].Name, check.Equals, "newPrefix")
+		c.Assert(result.Objects[1].Name, check.Equals, "newPrefix2")
+	}
+}
+
+// Tests validate overwriting of an existing object.
+func testObjectOverwriteWorks(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+
+	_, err = obj.PutObject("bucket", "object", int64(len("The list of parts was not in ascending order. The parts list must be specified in order by part number.")), bytes.NewBufferString("The list of parts was not in ascending order. The parts list must be specified in order by part number."), nil)
+	c.Assert(err, check.IsNil)
+
+	length := int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."))
+	_, err = obj.PutObject("bucket", "object", length, bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+	c.Assert(err, check.IsNil)
+
+	var bytesBuffer bytes.Buffer
+	err = obj.GetObject("bucket", "object", 0, length, &bytesBuffer)
+	c.Assert(err, check.IsNil)
+	c.Assert(string(bytesBuffer.Bytes()), check.Equals, "The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")
+}
+
+// Tests validate that bucket operation on non-existent bucket fails.
+func testNonExistantBucketOperations(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	_, err := obj.PutObject("bucket1", "object", int64(len("one")), bytes.NewBufferString("one"), nil)
+	c.Assert(err, check.Not(check.IsNil))
+	c.Assert(err.Error(), check.Equals, "Bucket not found: bucket1")
+}
+
+// Tests validate that recreation of the bucket fails.
+func testBucketRecreateFails(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("string")
+	c.Assert(err, check.IsNil)
+	err = obj.MakeBucket("string")
+	c.Assert(err, check.Not(check.IsNil))
+	c.Assert(err.Error(), check.Equals, "Bucket exists: string")
+}
+
+// Tests validate PutObject with subdirectory prefix.
+func testPutObjectInSubdir(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+
+	length := int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."))
+	_, err = obj.PutObject("bucket", "dir1/dir2/object", length, bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+	c.Assert(err, check.IsNil)
+
+	var bytesBuffer bytes.Buffer
+	err = obj.GetObject("bucket", "dir1/dir2/object", 0, length, &bytesBuffer)
+	c.Assert(err, check.IsNil)
+	c.Assert(len(bytesBuffer.Bytes()), check.Equals, len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."))
+}
+
+// Tests validate ListBuckets.
+func testListBuckets(c *check.C, create func() ObjectLayer) {
+	obj := create()
+
+	// test empty list.
+	buckets, err := obj.ListBuckets()
+	c.Assert(err, check.IsNil)
+	c.Assert(len(buckets), check.Equals, 0)
+
+	// add one and test exists.
+	err = obj.MakeBucket("bucket1")
+	c.Assert(err, check.IsNil)
+
+	buckets, err = obj.ListBuckets()
+	c.Assert(len(buckets), check.Equals, 1)
+	c.Assert(err, check.IsNil)
+
+	// add two and test exists.
+	err = obj.MakeBucket("bucket2")
+	c.Assert(err, check.IsNil)
+
+	buckets, err = obj.ListBuckets()
+	c.Assert(len(buckets), check.Equals, 2)
+	c.Assert(err, check.IsNil)
+
+	// add three and test exists + prefix.
+	err = obj.MakeBucket("bucket22")
+
+	buckets, err = obj.ListBuckets()
+	c.Assert(len(buckets), check.Equals, 3)
+	c.Assert(err, check.IsNil)
+}
+
+// Tests validate the order of result of ListBuckets.
+func testListBucketsOrder(c *check.C, create func() ObjectLayer) {
+	// if implementation contains a map, order of map keys will vary.
+	// this ensures they return in the same order each time.
+	for i := 0; i < 10; i++ {
+		obj := create()
+		// add one and test exists.
+		err := obj.MakeBucket("bucket1")
+		c.Assert(err, check.IsNil)
+		err = obj.MakeBucket("bucket2")
+		c.Assert(err, check.IsNil)
+		buckets, err := obj.ListBuckets()
+		c.Assert(err, check.IsNil)
+		c.Assert(len(buckets), check.Equals, 2)
+		c.Assert(buckets[0].Name, check.Equals, "bucket1")
+		c.Assert(buckets[1].Name, check.Equals, "bucket2")
+	}
+}
+
+// Tests validate that ListObjects operation on a non-existent bucket fails as expected.
+func testListObjectsTestsForNonExistantBucket(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	result, err := obj.ListObjects("bucket", "", "", "", 1000)
+	c.Assert(err, check.Not(check.IsNil))
+	c.Assert(result.IsTruncated, check.Equals, false)
+	c.Assert(len(result.Objects), check.Equals, 0)
+	c.Assert(err.Error(), check.Equals, "Bucket not found: bucket")
+}
+
+// Tests validate that GetObject fails on a non-existent bucket as expected.
+func testNonExistantObjectInBucket(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+
+	var bytesBuffer bytes.Buffer
+	err = obj.GetObject("bucket", "dir1", 0, 10, &bytesBuffer)
+	c.Assert(err, check.Not(check.IsNil))
+	switch err := err.(type) {
+	case ObjectNotFound:
+		c.Assert(err, check.ErrorMatches, "Object not found: bucket#dir1")
+	default:
+		c.Assert(err, check.Equals, "fails")
+	}
+}
+
+// Tests validate that GetObject on an existing directory fails as expected.
+func testGetDirectoryReturnsObjectNotFound(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+
+	_, err = obj.PutObject("bucket", "dir1/dir3/object", int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag."), nil)
+	c.Assert(err, check.IsNil)
+
+	var bytesBuffer bytes.Buffer
+	err = obj.GetObject("bucket", "dir1", 0, 10, &bytesBuffer)
+	switch err := err.(type) {
+	case ObjectNotFound:
+		c.Assert(err.Bucket, check.Equals, "bucket")
+		c.Assert(err.Object, check.Equals, "dir1")
+	default:
+		// force a failure with a line number.
+		c.Assert(err, check.Equals, "ObjectNotFound")
+	}
+
+	err = obj.GetObject("bucket", "dir1/", 0, 10, &bytesBuffer)
+	switch err := err.(type) {
+	case ObjectNameInvalid:
+		c.Assert(err.Bucket, check.Equals, "bucket")
+		c.Assert(err.Object, check.Equals, "dir1/")
+	default:
+		// force a failure with a line number.
+		c.Assert(err, check.Equals, "ObjectNotFound")
+	}
+}
+
+// Tests valdiate the default ContentType.
+func testDefaultContentType(c *check.C, create func() ObjectLayer) {
+	obj := create()
+	err := obj.MakeBucket("bucket")
+	c.Assert(err, check.IsNil)
+
+	// Test empty.
+	_, err = obj.PutObject("bucket", "one", int64(len("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")), bytes.NewBufferString("The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed."), nil)
+	c.Assert(err, check.IsNil)
+	objInfo, err := obj.GetObjectInfo("bucket", "one")
+	c.Assert(err, check.IsNil)
+	c.Assert(objInfo.ContentType, check.Equals, "application/octet-stream")
+}

pkg/minhttp/LICENSE.Facebook

@@ -1,30 +0,0 @@
-BSD License
-
-For grace software
-
-Copyright (c) 2015, Facebook, Inc. All rights reserved.
-
-Redistribution and use in source and binary forms, with or without modification,
-are permitted provided that the following conditions are met:
-
- * Redistributions of source code must retain the above copyright notice, this
-   list of conditions and the following disclaimer.
-
- * Redistributions in binary form must reproduce the above copyright notice,
-   this list of conditions and the following disclaimer in the documentation
-   and/or other materials provided with the distribution.
-
- * Neither the name Facebook nor the names of its contributors may be used to
-   endorse or promote products derived from this software without specific
-   prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

pkg/minhttp/LICENSE.Minio

@@ -1,202 +0,0 @@
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

pkg/minhttp/http_nix.go

@@ -1,214 +0,0 @@
-// +build linux darwin freebsd openbsd netbsd dragonfly
-
-/*
- * Minio Cloud Storage, (C) 2015 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-// Package minhttp provides easy to use graceful restart for a set of HTTP services
-//
-// This package is a fork from https://github.com/facebookgo/grace
-//
-// Re-licensing with Apache License 2.0, with code modifications
-package minhttp
-
-import (
-	"crypto/tls"
-	"net"
-	"net/http"
-	"os"
-	"os/signal"
-	"sync"
-	"syscall"
-	"time"
-
-	"github.com/facebookgo/httpdown"
-	"github.com/minio/minio/pkg/probe"
-)
-
-// An app contains one or more servers and their associated configuration.
-type app struct {
-	servers   []*http.Server
-	listeners []net.Listener
-	sds       []httpdown.Server
-	net       *minNet
-	errors    chan *probe.Error
-}
-
-// listen initailize listeners
-func (a *app) listen() *probe.Error {
-	for _, s := range a.servers {
-		l, err := a.net.Listen("tcp", s.Addr)
-		if err != nil {
-			return err.Trace()
-		}
-		if s.TLSConfig != nil {
-			l = tls.NewListener(l, s.TLSConfig)
-		}
-		a.listeners = append(a.listeners, l)
-	}
-	return nil
-}
-
-// serve start serving all listeners
-func (a *app) serve() {
-	h := &httpdown.HTTP{
-		StopTimeout: 10 * time.Second,
-		KillTimeout: 1 * time.Second,
-	}
-	for i, s := range a.servers {
-		a.sds = append(a.sds, h.Serve(s, a.listeners[i]))
-	}
-}
-
-// wait for http server to signal all requests that have been served
-func (a *app) wait() {
-	var wg sync.WaitGroup
-	wg.Add(len(a.sds) * 2) // Wait & Stop
-	go a.trapSignal(&wg)
-	for _, s := range a.sds {
-		go func(s httpdown.Server) {
-			defer wg.Done()
-			if err := s.Wait(); err != nil {
-				a.errors <- probe.NewError(err)
-			}
-		}(s)
-	}
-	wg.Wait()
-}
-
-// trapSignal wait on listed signals for pre-defined behaviors
-func (a *app) trapSignal(wg *sync.WaitGroup) {
-	ch := make(chan os.Signal, 10)
-	signal.Notify(ch, syscall.SIGTERM, syscall.SIGHUP)
-	for {
-		sig := <-ch
-		switch sig {
-		case syscall.SIGTERM:
-			// this ensures a subsequent TERM will trigger standard go behaviour of terminating
-			signal.Stop(ch)
-			// roll through all initialized http servers and stop them
-			for _, s := range a.sds {
-				go func(s httpdown.Server) {
-					defer wg.Done()
-					if err := s.Stop(); err != nil {
-						a.errors <- probe.NewError(err)
-					}
-				}(s)
-			}
-			return
-		case syscall.SIGHUP:
-			// we only return here if there's an error, otherwise the new process
-			// will send us a TERM when it's ready to trigger the actual shutdown.
-			if _, err := a.net.StartProcess(); err != nil {
-				a.errors <- err.Trace()
-			}
-		}
-	}
-}
-
-// ListenAndServe will serve the given http.Servers and will monitor for signals
-// allowing for graceful termination (SIGTERM) or restart (SIGUSR2/SIGHUP).
-func ListenAndServe(servers ...*http.Server) *probe.Error {
-	// get parent process id
-	ppid := os.Getppid()
-
-	a := &app{
-		servers:   servers,
-		listeners: make([]net.Listener, 0, len(servers)),
-		sds:       make([]httpdown.Server, 0, len(servers)),
-		net:       &minNet{},
-		errors:    make(chan *probe.Error, 1+(len(servers)*2)),
-	}
-
-	// Acquire Listeners
-	if err := a.listen(); err != nil {
-		return err.Trace()
-	}
-
-	// Start serving.
-	a.serve()
-
-	// Close the parent if we inherited and it wasn't init that started us.
-	if os.Getenv("LISTEN_FDS") != "" && ppid != 1 {
-		if err := syscall.Kill(ppid, syscall.SIGTERM); err != nil {
-			return probe.NewError(err)
-		}
-	}
-
-	waitdone := make(chan struct{})
-	go func() {
-		defer close(waitdone)
-		a.wait()
-		// communicate by sending not by closing a channel
-		waitdone <- struct{}{}
-	}()
-
-	select {
-	case err := <-a.errors:
-		if err == nil {
-			panic("unexpected nil error")
-		}
-		return err.Trace()
-	case <-waitdone:
-		return nil
-	}
-}
-
-// ListenAndServeLimited is similar to ListenAndServe but ratelimited with connLimit value
-func ListenAndServeLimited(connLimit int, servers ...*http.Server) *probe.Error {
-	// get parent process id
-	ppid := os.Getppid()
-
-	a := &app{
-		servers:   servers,
-		listeners: make([]net.Listener, 0, len(servers)),
-		sds:       make([]httpdown.Server, 0, len(servers)),
-		net:       &minNet{connLimit: connLimit},
-		errors:    make(chan *probe.Error, 1+(len(servers)*2)),
-	}
-
-	// Acquire Listeners
-	if err := a.listen(); err != nil {
-		return err.Trace()
-	}
-
-	// Start serving.
-	a.serve()
-
-	// Close the parent if we inherited and it wasn't init that started us.
-	if os.Getenv("LISTEN_FDS") != "" && ppid != 1 {
-		if err := syscall.Kill(ppid, syscall.SIGTERM); err != nil {
-			return probe.NewError(err)
-		}
-	}
-
-	waitdone := make(chan struct{})
-	go func() {
-		defer close(waitdone)
-		a.wait()
-		// communicate by sending not by closing a channel
-		waitdone <- struct{}{}
-	}()
-
-	select {
-	case err := <-a.errors:
-		if err == nil {
-			panic("unexpected nil error")
-		}
-		return err.Trace()
-	case <-waitdone:
-		return nil
-	}
-}

pkg/minhttp/http_windows.go

@@ -1,214 +0,0 @@
-// +build windows
-
-/*
- * Minio Cloud Storage, (C) 2015 Minio, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-// Package minhttp provides easy to use graceful restart for a set of HTTP services
-//
-// This package is a fork from https://github.com/facebookgo/grace
-//
-// Re-licensing with Apache License 2.0, with code modifications
-package minhttp
-
-import (
-	"crypto/tls"
-	"net"
-	"net/http"
-	"os"
-	"os/signal"
-	"sync"
-	"syscall"
-	"time"
-
-	"github.com/facebookgo/httpdown"
-	"github.com/minio/minio/pkg/probe"
-)
-
-// An app contains one or more servers and their associated configuration.
-type app struct {
-	servers   []*http.Server
-	listeners []net.Listener
-	sds       []httpdown.Server
-	net       *minNet
-	errors    chan *probe.Error
-}
-
-// listen initailize listeners
-func (a *app) listen() *probe.Error {
-	for _, s := range a.servers {
-		l, err := a.net.Listen("tcp", s.Addr)
-		if err != nil {
-			return err.Trace()
-		}
-		if s.TLSConfig != nil {
-			l = tls.NewListener(l, s.TLSConfig)
-		}
-		a.listeners = append(a.listeners, l)
-	}
-	return nil
-}
-
-// serve start serving all listeners
-func (a *app) serve() {
-	h := &httpdown.HTTP{
-		StopTimeout: 10 * time.Second,
-		KillTimeout: 1 * time.Second,
-	}
-	for i, s := range a.servers {
-		a.sds = append(a.sds, h.Serve(s, a.listeners[i]))
-	}
-}
-
-// wait for http server to signal all requests that have been served
-func (a *app) wait() {
-	var wg sync.WaitGroup
-	wg.Add(len(a.sds) * 2) // Wait & Stop
-	go a.trapSignal(&wg)
-	for _, s := range a.sds {
-		go func(s httpdown.Server) {
-			defer wg.Done()
-			if err := s.Wait(); err != nil {
-				a.errors <- probe.NewError(err)
-			}
-		}(s)
-	}
-	wg.Wait()
-}
-
-// trapSignal wait on listed signals for pre-defined behaviors
-func (a *app) trapSignal(wg *sync.WaitGroup) {
-	ch := make(chan os.Signal, 10)
-	signal.Notify(ch, syscall.SIGTERM, syscall.SIGHUP)
-	for {
-		sig := <-ch
-		switch sig {
-		case syscall.SIGTERM:
-			// this ensures a subsequent TERM will trigger standard go behaviour of terminating
-			signal.Stop(ch)
-			// roll through all initialized http servers and stop them
-			for _, s := range a.sds {
-				go func(s httpdown.Server) {
-					defer wg.Done()
-					if err := s.Stop(); err != nil {
-						a.errors <- probe.NewError(err)
-					}
-				}(s)
-			}
-			return
-		case syscall.SIGHUP:
-			// we only return here if there's an error, otherwise the new process
-			// will send us a TERM when it's ready to trigger the actual shutdown.
-			if _, err := a.net.StartProcess(); err != nil {
-				a.errors <- err.Trace()
-			}
-		}
-	}
-}
-
-// ListenAndServe will serve the given http.Servers and will monitor for signals
-// allowing for graceful termination (SIGTERM) or restart (SIGUSR2/SIGHUP).
-func ListenAndServe(servers ...*http.Server) *probe.Error {
-	// get parent process id
-	ppid := os.Getppid()
-
-	a := &app{
-		servers:   servers,
-		listeners: make([]net.Listener, 0, len(servers)),
-		sds:       make([]httpdown.Server, 0, len(servers)),
-		net:       &minNet{},
-		errors:    make(chan *probe.Error, 1+(len(servers)*2)),
-	}
-
-	// Acquire Listeners
-	if err := a.listen(); err != nil {
-		return err.Trace()
-	}
-
-	// Start serving.
-	a.serve()
-
-	// Close the parent if we inherited and it wasn't init that started us.
-	if os.Getenv("LISTEN_FDS") != "" && ppid != 1 {
-		if err := terminateProcess(ppid, 1); err != nil {
-			return probe.NewError(err)
-		}
-	}
-
-	waitdone := make(chan struct{})
-	go func() {
-		defer close(waitdone)
-		a.wait()
-		// communicate by sending not by closing a channel
-		waitdone <- struct{}{}
-	}()
-
-	select {
-	case err := <-a.errors:
-		if err == nil {
-			panic("unexpected nil error")
-		}
-		return err.Trace()
-	case <-waitdone:
-		return nil
-	}
-}
-
-// ListenAndServeLimited is similar to ListenAndServe but ratelimited with connLimit value
-func ListenAndServeLimited(connLimit int, servers ...*http.Server) *probe.Error {
-	// get parent process id
-	ppid := os.Getppid()
-
-	a := &app{
-		servers:   servers,
-		listeners: make([]net.Listener, 0, len(servers)),
-		sds:       make([]httpdown.Server, 0, len(servers)),
-		net:       &minNet{connLimit: connLimit},
-		errors:    make(chan *probe.Error, 1+(len(servers)*2)),
-	}
-
-	// Acquire Listeners
-	if err := a.listen(); err != nil {
-		return err.Trace()
-	}
-
-	// Start serving.
-	a.serve()
-
-	// Close the parent if we inherited and it wasn't init that started us.
-	if os.Getenv("LISTEN_FDS") != "" && ppid != 1 {
-		if err := terminateProcess(ppid, 1); err != nil {
-			return probe.NewError(err)
-		}
-	}
-
-	waitdone := make(chan struct{})
-	go func() {
-		defer close(waitdone)
-		a.wait()
-		// communicate by sending not by closing a channel
-		waitdone <- struct{}{}
-	}()
-
-	select {
-	case err := <-a.errors:
-		if err == nil {
-			panic("unexpected nil error")
-		}
-		return err.Trace()
-	case <-waitdone:
-		return nil
-	}
-}