Compare commits

..

41 Commits

Author SHA1 Message Date
Harshavardhana 6df05e489d Set Read/Write timeouts only for net.Conn not http.Server (#7431)
Fixes #7425
2019-03-27 22:10:06 +05:30
Harshavardhana 4a698c731b HealObjects should remove objects without quorum (#7407)
This PR adds a way to list objects without quorum
such that they can purged by `mc admin heal --remove`
2019-03-26 14:57:44 -07:00
Harshavardhana 9629de8230 Add proper context based logging when bitrot stream calls fail (#7415) 2019-03-26 13:59:33 -07:00
Harshavardhana 0250f7de67 Cleanup stale multipart uploads older than 3 days (#7424)
Fixes #6627
2019-03-25 13:41:05 -07:00
kannappanr 7154b8a568 Error log: Correct error type in anonymous mode (#7414)
Currently message is set to error type value.
Message field is not used in error logs. it is used only in the case of info logs.

This PR sets error message field to store error type correctly.
2019-03-25 13:40:08 -07:00
Anis Elleuch 8689ec258b Don't decrypt ETag in validation when source is SSEC multipart (#7423)
Copying an encrypted SSEC object when this latter is uploaded using
multipart mechanism was failing because ETag in case of encrypted
multipart upload is not encrypted.

This PR fixes the behavior.
2019-03-25 12:17:31 -07:00
Krishnan Parthasarathi aac9e2a7dd Return deploymentID in ServerInfo REST call (#7422)
Makes deploymentID information uniform in distributed setup
2019-03-25 11:55:28 -07:00
Harshavardhana 719d21efd8 Generate coverage across all sub-dirs (#7416) 2019-03-25 11:54:14 -07:00
Harshavardhana e0a87e96de Populate host value from GetSourceIP directly (#7417) 2019-03-25 11:45:42 -07:00
Harshavardhana 0a44e70177 Change findEndpoint to return errors for Scanlines (#7390) 2019-03-25 13:31:02 +05:30
Rushan 22b4fe0a51 Update browser UI with new logo and colors (#7408) 2019-03-23 09:57:09 -07:00
Praveen raj Mani 89e45d0695 Restart process should use the current process' pid (#7373)
This fixes varying pids for server-respawns. And avoids duplicate process
creating multiple pids when the server restart signal is triggered with
service restart enabled.

Fixes #7350
2019-03-20 22:20:30 -07:00
poornas 8e1e701d35 Azure:ETag returned by ListObjects to be consistent with GetObjectInfo (#7301) 2019-03-20 18:11:46 -07:00
kannappanr 87cf51d5ab unused code: Remove LoadCredentials function (#7369)
It is required to set the environment variable in the case of distributed
minio. LoadCredentials is used to notify peers of the change and will not work if
environment variable is set. so, this function will never be called.
2019-03-20 18:09:57 -07:00
Minio Trusted 8af45f0135 Update yaml files to latest version RELEASE.2019-03-20T22-38-47Z 2019-03-20 22:49:06 +00:00
Harshavardhana a9032b52b8 Change storageRESTTimeout to 1minute (#7398) 2019-03-20 13:20:09 -07:00
Harshavardhana c184038b6a Add proper custom errors object creations (#7387)
In scenario 1

```
- bucket/object-prefix
- bucket/object-prefix/object
```

Server responds with `XMinioParentIsObject`

In scenario 2

```
- bucket/object-prefix/object
- bucket/object-prefix
```

Server responds with `XMinioObjectExistsAsDirectory`

Fixes #6566
2019-03-20 13:06:53 -07:00
poornas 12b79d9f3b Remove duplicate error in switch case. (#7381)
Fixes: #7380

crypto.ErrInvalidCustomerKey was being handled twice in toAPIErrorCode()
2019-03-19 17:21:05 -07:00
Sidhartha Mani 6bc0de2a75 add go modules file and start running go 1.11 style builds (#7354) 2019-03-19 13:50:58 -07:00
Krishnan Parthasarathi 8a77a298f2 Add deploymentID to ServerInfo (#7372) 2019-03-19 16:12:24 +05:30
Harshavardhana 328eb74cbb Fix regression in peer clients in TLS setups (#7391)
Regression was introduced in eb69c4f946
2019-03-19 09:44:49 +05:30
zy 73be3ed0ca format import style (#7383) 2019-03-18 13:07:58 -07:00
Kirill Motkov 3d29ab4059 Rewrite if-else chains to switch statements (#7382) 2019-03-18 07:46:20 -07:00
Harshavardhana 6702d23d52 Simplify ReadFileStream closer, make sure to flush all HTTP responses (#7374) 2019-03-18 10:50:26 +05:30
poornas 1011d21416 Fix credential parsing in signature v4 (#7377)
Fixes #7376
2019-03-16 22:45:42 -07:00
Kirill Motkov 85c5acc088 fix staticcheck warning (#7378) 2019-03-16 22:44:43 -07:00
kannappanr eb69c4f946 Use REST api for inter node communication (#7205) 2019-03-14 16:27:31 -07:00
Anis Elleuch facbd653ba Add normal/deep type of heal scanning (#7251)
Healing scan used to read all objects parts to check for bitrot
checksum. This commit will add a quicker way of healing scan
by only checking if parts are actually present in disks or not.
2019-03-14 13:08:51 -07:00
Harshavardhana 233824bf92 Configure http2 with higher maxconcurrent streams (#7363)
This value is needed for Minio's internode communication,
read the meaning of this value as per the HTTP 2.0 spec

https://http2.github.io/http2-spec/#rfc.section.5.1.2
2019-03-14 11:57:35 -07:00
poornas a0ee7be050 update link on README (#7361) 2019-03-14 15:18:12 +05:30
Harshavardhana 91d85a0d53 Fix stale locks held by SelectParquet API (#7364)
Vendorize upstream parquet-go to fix this issue.
2019-03-13 20:33:18 -07:00
Harshavardhana 7079abc931 Implement HealObjects API to simplify healing (#7351) 2019-03-13 17:35:09 -07:00
Minio Trusted 8377d00574 Update yaml files to latest version RELEASE.2019-03-13T21-59-47Z 2019-03-13 22:06:26 +00:00
Harshavardhana 285c09fe6b Support buckets with '.' with etcd+coreDNS (#7353)
Fixes #7340
2019-03-12 17:57:08 -07:00
kannappanr ce4563370c Distributed: Allow healing if all disks are on root partitions (#7358)
If all the disks are on root partitions in distributed mode, consider it
to be a test setup and allow healing to proceed.

Fixes #7346
2019-03-12 16:47:06 -07:00
Aditya Manthramurthy e463386921 Add JSON Path expression evaluation support (#7315)
- Includes support for FROM clause JSON path
2019-03-09 08:13:37 -08:00
Aditya Manthramurthy b296b3cf8b Skip non-bucket dns entry in federated bucket list (#7339) 2019-03-08 14:22:58 -08:00
Harshavardhana 0b96ad4fdc http2 throws custom error Content-Length shorter handle it (#7334)
We should internally handle when http2 input stream has smaller
content than its content-length header

Upstream issue reported https://github.com/golang/go/issues/30648

This a change which we need to handle internally until Go fixes it
correctly, till now our code doesn't expect a custom error to be returned.
2019-03-07 16:11:28 -08:00
Aditya Manthramurthy f4879ed96d Use jstream to serialize records to JSON format in S3Select (#7318)
- Also, switch to jstream to generate internal record representation
  from CSV/JSON readers

- This fixes a bug in which JSON output objects have their keys
  reversed from the order they are specified in the Select columns.

- Also includes a fix for tests.
2019-03-07 00:20:10 -08:00
Aditya Manthramurthy f97a33a63f Downgrade node temporarily (#7333)
To fix jest related build failure as recommended at https://github.com/facebook/jest/issues/8069#issuecomment-470307590
2019-03-06 16:05:49 -08:00
Minio Trusted 6387407e79 Update yaml files to latest version RELEASE.2019-03-06T22-47-10Z 2019-03-06 22:56:46 +00:00
144 changed files with 7067 additions and 3121 deletions
+2 -1
View File
@@ -17,6 +17,7 @@ matrix:
env:
- ARCH=x86_64
- CGO_ENABLED=0
- GO111MODULE=on
go: 1.11.4
script:
- make
@@ -44,7 +45,7 @@ before_script:
- if [[ "${TRAVIS_OS_NAME}" == "linux" ]]; then sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'; fi
before_install:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then nvm install stable ; fi
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then nvm install 11.10.1 ; fi
after_success:
- bash <(curl -s https://codecov.io/bash)
+4 -3
View File
@@ -7,6 +7,9 @@ COPY . /go/src/github.com/minio/minio
WORKDIR /go/src/github.com/minio/minio
RUN apt-get update && apt-get install -y jq
ENV GO111MODULE=on
RUN git config --global http.cookiefile /gitcookie/.gitcookie
RUN make
RUN bash -c 'diff -au <(gofmt -s -d cmd) <(printf "")'
@@ -28,6 +31,7 @@ RUN groupadd -g 999 ci && \
RUN chown -R ci:ci /go
RUN chown -R ci:ci /home/ci
RUN chmod -R a+rw /go
USER ci
@@ -58,9 +62,6 @@ COPY --from=0 /go/src/github.com/minio/minio/minio ./minio
COPY buildscripts/gateway-tests.sh ./gateway-tests.sh
RUN apt-get update && apt-get install -y git wget jq curl dnsmasq
RUN wget https://dl.google.com/go/go1.11.4.linux-amd64.tar.gz && \
tar -C /usr/local -xzf go1.11.4.linux-amd64.tar.gz
ENV DEBIAN_FRONTEND noninteractive
ENV LANG C.UTF-8
ENV GOROOT /usr/local/go
+4 -3
View File
@@ -13,9 +13,10 @@ checks:
@(env bash $(PWD)/buildscripts/checkgopath.sh)
getdeps:
@echo "Installing golint" && go get -u golang.org/x/lint/golint
@echo "Installing staticcheck" && go get -u honnef.co/go/tools/...
@echo "Installing misspell" && go get -u github.com/client9/misspell/cmd/misspell
@mkdir -p $(GOPATH)/bin
@echo "Installing golint" && which golint || go get -u golang.org/x/lint/golint
@echo "Installing staticcheck" && which staticcheck || wget --quiet -O $(GOPATH)/bin/staticcheck https://github.com/dominikh/go-tools/releases/download/2019.1/staticcheck_linux_amd64 && chmod +x $(GOPATH)/bin/staticcheck
@echo "Installing misspell" && which misspell || wget --quiet https://github.com/client9/misspell/releases/download/v0.3.4/misspell_0.3.4_linux_64bit.tar.gz && tar xvf misspell_0.3.4_linux_64bit.tar.gz && mv misspell $(GOPATH)/bin/misspell && chmod +x $(GOPATH)/bin/misspell && rm -r misspell_0.3.4_linux_64bit.tar.gz
crosscompile:
@(env bash $(PWD)/buildscripts/cross-compile.sh)
+1 -1
View File
@@ -4,7 +4,7 @@
height: 100%;
top: 0;
left: 0;
background: #32393F;
background: #002a37;
z-index: 100;
transition: opacity 200ms;
-webkit-transition: opacity 200ms;
Binary file not shown.

Before

Width:  |  Height:  |  Size: 501 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

+50
View File
@@ -0,0 +1,50 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="469px" height="60px" viewBox="0 0 469 60" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Generator: Sketch 48.2 (47327) - http://www.bohemiancoding.com/sketch -->
<title>Untitled</title>
<desc>Created with Sketch.</desc>
<defs></defs>
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<path d="M27.9171869,0.94921875 L5.01068171,0.94921875 C2.24925796,0.94921875 0.0106817104,3.187795 0.0106817104,5.94921875 L0.0106817104,54.9836742 C0.0106817104,57.7450979 2.24925796,59.9836742 5.01068171,59.9836742 L43.0384854,59.9836742 C45.7999092,59.9836742 48.0384854,57.7450979 48.0384854,54.9836742 L48.0384854,21.4784647 C48.0384854,20.1754261 47.5298023,18.9238644 46.6207587,17.9902963 L31.4994602,2.46105043 C30.558243,1.49444073 29.2663438,0.94921875 27.9171869,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M87.9171869,0.94921875 L65.0106817,0.94921875 C62.249258,0.94921875 60.0106817,3.187795 60.0106817,5.94921875 L60.0106817,54.9836742 C60.0106817,57.7450979 62.249258,59.9836742 65.0106817,59.9836742 L103.038485,59.9836742 C105.799909,59.9836742 108.038485,57.7450979 108.038485,54.9836742 L108.038485,21.4784647 C108.038485,20.1754261 107.529802,18.9238644 106.620759,17.9902963 L91.4994602,2.46105043 C90.558243,1.49444073 89.2663438,0.94921875 87.9171869,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M147.917187,0.94921875 L125.010682,0.94921875 C122.249258,0.94921875 120.010682,3.187795 120.010682,5.94921875 L120.010682,54.9836742 C120.010682,57.7450979 122.249258,59.9836742 125.010682,59.9836742 L163.038485,59.9836742 C165.799909,59.9836742 168.038485,57.7450979 168.038485,54.9836742 L168.038485,21.4784647 C168.038485,20.1754261 167.529802,18.9238644 166.620759,17.9902963 L151.49946,2.46105043 C150.558243,1.49444073 149.266344,0.94921875 147.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M207.917187,0.94921875 L185.010682,0.94921875 C182.249258,0.94921875 180.010682,3.187795 180.010682,5.94921875 L180.010682,54.9836742 C180.010682,57.7450979 182.249258,59.9836742 185.010682,59.9836742 L223.038485,59.9836742 C225.799909,59.9836742 228.038485,57.7450979 228.038485,54.9836742 L228.038485,21.4784647 C228.038485,20.1754261 227.529802,18.9238644 226.620759,17.9902963 L211.49946,2.46105043 C210.558243,1.49444073 209.266344,0.94921875 207.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M267.917187,0.94921875 L245.010682,0.94921875 C242.249258,0.94921875 240.010682,3.187795 240.010682,5.94921875 L240.010682,54.9836742 C240.010682,57.7450979 242.249258,59.9836742 245.010682,59.9836742 L283.038485,59.9836742 C285.799909,59.9836742 288.038485,57.7450979 288.038485,54.9836742 L288.038485,21.4784647 C288.038485,20.1754261 287.529802,18.9238644 286.620759,17.9902963 L271.49946,2.46105043 C270.558243,1.49444073 269.266344,0.94921875 267.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M327.917187,0.94921875 L305.010682,0.94921875 C302.249258,0.94921875 300.010682,3.187795 300.010682,5.94921875 L300.010682,54.9836742 C300.010682,57.7450979 302.249258,59.9836742 305.010682,59.9836742 L343.038485,59.9836742 C345.799909,59.9836742 348.038485,57.7450979 348.038485,54.9836742 L348.038485,21.4784647 C348.038485,20.1754261 347.529802,18.9238644 346.620759,17.9902963 L331.49946,2.46105043 C330.558243,1.49444073 329.266344,0.94921875 327.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M387.917187,0.94921875 L365.010682,0.94921875 C362.249258,0.94921875 360.010682,3.187795 360.010682,5.94921875 L360.010682,54.9836742 C360.010682,57.7450979 362.249258,59.9836742 365.010682,59.9836742 L403.038485,59.9836742 C405.799909,59.9836742 408.038485,57.7450979 408.038485,54.9836742 L408.038485,21.4784647 C408.038485,20.1754261 407.529802,18.9238644 406.620759,17.9902963 L391.49946,2.46105043 C390.558243,1.49444073 389.266344,0.94921875 387.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<path d="M447.917187,0.94921875 L425.010682,0.94921875 C422.249258,0.94921875 420.010682,3.187795 420.010682,5.94921875 L420.010682,54.9836742 C420.010682,57.7450979 422.249258,59.9836742 425.010682,59.9836742 L463.038485,59.9836742 C465.799909,59.9836742 468.038485,57.7450979 468.038485,54.9836742 L468.038485,21.4784647 C468.038485,20.1754261 467.529802,18.9238644 466.620759,17.9902963 L451.49946,2.46105043 C450.558243,1.49444073 449.266344,0.94921875 447.917187,0.94921875 Z" id="Path" fill="#2E3D45"></path>
<g id="excel" transform="translate(434.000000, 28.000000)" fill="#617A8B" fill-rule="nonzero">
<rect id="Rectangle-2" x="0" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="0" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="0" y="10.3086611" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="6.99999999" y="10.3086611" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="0" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="5.15433056" width="5" height="3.09259835"></rect>
<rect id="Rectangle-2" x="14" y="10.3086611" width="5" height="3.09259835"></rect>
</g>
<g id="folder" transform="translate(14.000000, 25.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M7.99999996,0 L2,0 C0.900000087,0 0.00999998996,0.900000093 0.00999998996,2.00000001 L0,14 C0,15.1000003 0.900000087,16 2,16 L18,16 C19.1000002,16 20,15.1000003 20,14 L20,4.00000001 C20,2.90000009 19.1000002,2.00000001 18,2.00000001 L9.99999996,2.00000001 L7.99999996,0 Z" id="Shape"></path>
</g>
<g id="image" transform="translate(72.000000, 24.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M13.590909,-3.76363638e-07 L9.6704545,5.22727238 L12.6500002,9.19999976 L10.9772726,10.4545451 C9.21045494,8.10227239 6.27272722,4.18181782 6.27272722,4.18181782 L-6.27273216e-08,12.5454541 L22.9999999,12.5454541 L13.590909,-3.76363638e-07 Z" id="Shape"></path>
</g>
<g id="pdf" transform="translate(135.000000, 22.500000)" fill="#617A8B" fill-rule="nonzero">
<path d="M12.9434813,11.7078193 C11.5243518,10.2886898 9.92783089,8.15999533 8.50870126,6.2086921 C8.86348367,4.61217127 9.04087488,3.37043286 9.04087488,2.66086804 C9.04087488,-0.886956014 4.78348601,-0.886956014 4.78348601,2.66086804 C4.78348601,3.37043286 5.49305082,4.61217127 6.73478924,6.5634745 C6.20261563,9.04695135 5.49305082,12.0626018 4.6060948,14.5460787 C2.83218278,15.2556434 1.59044436,16.1425995 0.880879547,16.6747731 C-0.00607646688,17.5617291 -0.183467669,18.6260763 0.171314736,19.5130323 C0.526097141,20.3999883 1.41305316,20.932162 2.30000917,20.932162 C2.83218278,20.932162 3.36435639,20.7547707 3.71913879,20.3999883 C3.89652999,20.2225971 4.6060948,19.5130323 5.84783322,15.6104258 C8.15391886,14.7234698 10.6373957,13.8365138 12.4113077,13.3043402 C14.0078286,14.7234698 15.249567,15.4330347 16.4913054,15.4330347 C17.7330438,15.4330347 18.6199998,14.5460787 18.6199998,13.4817314 C18.6199998,12.7721666 18.2652175,12.239993 17.5556526,11.7078193 C16.8460878,11.353037 15.9591318,11.1756458 14.7173934,11.1756458 C14.362611,11.353037 13.8304373,11.353037 12.9434813,11.7078193 Z M2.47740038,19.1582499 C2.30000917,19.3356411 2.30000917,19.3356411 2.30000917,19.3356411 C2.12261797,19.3356411 1.94522676,19.1582499 1.76783556,18.8034675 C1.59044436,18.6260763 1.59044436,18.0939027 2.12261797,17.7391203 C2.30000917,17.5617291 2.83218278,17.2069466 3.89652999,16.6747731 C3.18696518,18.2712939 2.65479157,18.9808587 2.47740038,19.1582499 Z M6.38000684,2.48347683 C6.38000684,1.77391203 6.73478924,1.41912963 6.73478924,1.41912963 C6.73478924,1.41912963 7.08957165,1.77391203 7.08957165,2.48347683 C7.08957165,2.83825924 7.08957165,3.37043286 6.91218044,4.07999767 C6.55739803,3.19304165 6.38000684,2.66086804 6.38000684,2.48347683 Z M6.55739803,13.6591226 C7.08957165,11.8852106 7.62174525,10.1112986 7.97652766,8.33738654 C9.04087488,9.75651615 10.1052221,10.9982546 11.1695693,12.0626018 C9.75043969,12.5947754 8.15391886,12.9495579 6.55739803,13.6591226 Z M17.023479,13.6591226 C17.023479,13.8365138 16.8460878,14.013905 16.4913054,14.013905 C16.3139142,14.013905 15.6043494,13.8365138 14.5400021,12.9495579 C14.7173934,12.9495579 14.7173934,12.9495579 14.8947846,12.9495579 C15.9591318,12.9495579 16.6686966,13.126949 16.8460878,13.3043402 C16.8460878,13.4817314 17.023479,13.6591226 17.023479,13.6591226 Z" id="Shape"></path>
</g>
<g id="video" transform="translate(196.000000, 24.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M13.1249998,-1.68750002e-07 L13.1249998,1.87499983 L11.2499998,1.87499983 L11.2499998,-1.68750002e-07 L3.74999978,-1.68750002e-07 L3.74999978,1.87499983 L1.87499978,1.87499983 L1.87499978,-1.68750002e-07 L-2.25000019e-07,-1.68750002e-07 L-2.25000019e-07,16.8749998 L1.87499978,16.8749998 L1.87499978,14.9999998 L3.74999978,14.9999998 L3.74999978,16.8749998 L11.2499998,16.8749998 L11.2499998,14.9999998 L13.1249998,14.9999998 L13.1249998,16.8749998 L14.9999998,16.8749998 L14.9999998,-1.68750002e-07 L13.1249998,-1.68750002e-07 Z M3.74999955,13.1249991 L1.87499967,13.1249991 L1.87499967,11.2499992 L3.74999955,11.2499992 L3.74999955,13.1249991 Z M3.74999955,9.37499927 L1.87499967,9.37499927 L1.87499967,7.49999938 L3.74999955,7.49999938 L3.74999955,9.37499927 Z M3.74999955,5.62499949 L1.87499967,5.62499949 L1.87499967,3.74999961 L3.74999955,3.74999961 L3.74999955,5.62499949 Z M13.124999,13.1249991 L11.2499991,13.1249991 L11.2499991,11.2499992 L13.124999,11.2499992 L13.124999,13.1249991 Z M13.124999,9.37499927 L11.2499991,9.37499927 L11.2499991,7.49999938 L13.124999,7.49999938 L13.124999,9.37499927 Z M13.124999,5.62499949 L11.2499991,5.62499949 L11.2499991,3.74999961 L13.124999,3.74999961 L13.124999,5.62499949 Z" id="Shape"></path>
</g>
<g id="audio" transform="translate(253.000000, 25.000000)" fill="#617A8B" fill-rule="nonzero">
<path d="M16.6899956,0.302885892 C16.4835716,0.100936704 16.2326065,0 15.9375267,0 C15.8194481,0 15.7159842,0.0145115457 15.6276386,0.0433073022 L6.41930364,2.81258763 C6.19803256,2.88469069 6.01719345,3.01271813 5.87698015,3.1965942 C5.73684439,3.38058392 5.66671835,3.58427601 5.66671835,3.80789779 L5.66671835,14.2686004 C5.02469108,13.9872357 4.31641429,13.8465532 3.54169413,13.8465532 C3.20970997,13.8465532 2.8536527,13.8844045 2.47371613,13.9599933 C2.09358575,14.0355821 1.71171093,14.1509926 1.32813045,14.3060728 C0.944356146,14.4611532 0.627141486,14.6793946 0.376215114,14.9604564 C0.125366273,15.2417075 0,15.5628561 0,15.9233335 C0,16.2840761 0.125366273,16.6048457 0.376215114,16.8862105 C0.627141486,17.1673102 0.944317382,17.3855517 1.32813045,17.540594 C1.71186599,17.6956744 2.09374081,17.810971 2.47371613,17.8866736 C2.8536527,17.9622625 3.20970997,18 3.54169413,18 C3.87367828,18 4.22961926,17.9622625 4.60967212,17.8866736 C4.9898025,17.810971 5.37152225,17.6956744 5.7552578,17.540594 C6.13887704,17.3855137 6.4560917,17.1672722 6.70694055,16.8862105 C6.95802198,16.6048457 7.08334949,16.2840381 7.08334949,15.9233335 L7.08334949,8.25381104 L15.5834076,5.69004149 L15.5834076,11.4990929 C14.9412253,11.2178795 14.2330648,11.0770455 13.4583446,11.0770455 C13.1263217,11.0770455 12.7702257,11.1147831 12.3902891,11.1904856 C12.0101975,11.2660745 11.6285553,11.3814848 11.2447422,11.5365651 C10.861123,11.6916454 10.543792,11.909887 10.2927493,12.1912139 C10.0420168,12.4722 9.91626286,12.7932726 9.91626286,13.1538259 C9.91626286,13.5145685 10.0420168,13.8353381 10.2927493,14.1167028 C10.543792,14.3979541 10.8610842,14.6161956 11.2447422,14.7710864 C11.6284002,14.9261668 12.0101975,15.041577 12.3902891,15.1173175 C12.7702257,15.1930201 13.126283,15.230606 13.4583446,15.230606 C13.7902513,15.230606 14.1463085,15.1930201 14.5262451,15.1173175 C14.9062204,15.041577 15.2879788,14.9261668 15.6719083,14.7710864 C16.0555662,14.6161576 16.3726259,14.3979161 16.6236297,14.1167028 C16.8747888,13.8353381 17,13.5145305 17,13.1538259 L17,1.03820069 C16.9998837,0.749788454 16.8969236,0.504721411 16.6899956,0.302885892 Z" id="Shape"></path>
</g>
<g id="code" transform="translate(313.000000, 27.000000)" fill="#617A8B" fill-rule="nonzero">
<polygon id="Shape" transform="translate(17.091428, 6.857142) scale(-1, 1) translate(-17.091428, -6.857142) " points="21.3257133 1.61142788 19.7142851 -4.11428573e-07 12.8571425 6.85714204 19.7142851 13.7142845 21.3257133 12.1028561 16.091428 6.85714204"></polygon>
<polygon id="Shape" points="8.46857015 1.61142788 6.8571419 -4.11428573e-07 -5.48571475e-07 6.85714204 6.8571419 13.7142845 8.46857015 12.1028561 3.23428485 6.85714204"></polygon>
</g>
<g id="presentation" transform="translate(374.464844, 23.500000)" fill="#617A8B" fill-rule="nonzero">
<path d="M8.16304337,0.383534587 L8.16304337,17.4406771 C3.83905758,17.0142486 0.48732923,13.3554907 0.48732923,8.91210585 C0.48732923,4.468721 3.83905799,0.809963151 8.16304337,0.383534587 Z M9.89434308,0.383534587 L9.89434308,8.05071991 L17.5444718,8.05071991 C17.1436294,4.00817738 13.9283578,0.784376849 9.89434308,0.383534587 Z M9.89434308,9.77349173 L9.89434308,17.4406771 C13.9368865,17.0398348 17.1436278,13.8160343 17.5444718,9.77349173 L9.89434308,9.77349173 Z" id="Shape"></path>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 14 KiB

+4 -8
View File
@@ -1,16 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="302px" height="579px" viewBox="0 0 302 579" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<svg width="93px" height="187px" viewBox="0 0 93 187" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Generator: Sketch 48.2 (47327) - http://www.bohemiancoding.com/sketch -->
<title>Untitled 2</title>
<title>logo</title>
<desc>Created with Sketch.</desc>
<defs></defs>
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<g id="Minio_Logo_White" fill-rule="nonzero">
<g id="Group">
<path d="M225.835,28.882 C225.835,28.882 282.569,120.259 301.531,152.735 C301.837,153.264 301.707,153.963 301.251,154.419 C300.676,154.994 299.81,155.056 299.24,154.481 L201.484,52.243 L225.835,28.882 Z" id="Shape" fill="#F15A29"></path>
<path d="M58.706,361.854 C75.299,326.75 97.681,294.989 125.38,267.29 C135.915,256.76 147.036,247.002 158.701,238.037 L158.701,310.385 L58.706,361.854 Z M0.971,428.672 L158.7,348.375 L158.7,531.932 L194.244,578.623 L194.244,330.092 L215.833,318.987 C225.778,313.991 235.105,307.343 243.402,299.041 C284.517,257.931 285.17,191.574 245.226,149.749 L163.742,64.855 C160.493,61.383 158.788,56.838 158.938,52.06 C159.088,47.251 161.099,42.784 164.597,39.488 C171.811,32.704 183.305,33.161 190.088,40.369 L201.484,52.247 L225.835,28.881 C197.064,-8.431 161.861,-3.746 141.47,14.92 C131.417,24.383 125.644,37.198 125.208,51.003 C124.773,64.829 129.753,77.981 139.221,88.04 L139.33,88.159 L221.239,173.483 C248.176,202.151 247.637,247.235 219.617,275.255 C215.248,279.623 210.46,283.324 205.387,286.36 L194.245,292.091 L194.245,176.126 C160.167,193.688 129.09,216.251 101.692,243.655 C67.568,277.774 40.776,317.516 22.058,361.772 C12.875,383.479 5.837,405.829 0.971,428.672 Z" id="Shape" fill="#FFFFFF"></path>
<polygon id="Shape" fill="#BCBEC0" points="194.246 330.09 194.246 367.609 158.701 385.648 158.701 348.373"></polygon>
</g>
<g id="logo" transform="translate(0.187500, -0.683594)" fill="#FFFFFF" fill-rule="nonzero">
<path d="M91.49,46.551 C86.7827023,38.7699609 82.062696,30.9966172 77.33,23.231 C74.87,19.231 72.33,15.231 69.88,11.231 C69.57,10.731 69.18,10.291 68.88,9.831 C64.35,2.931 55.44,-1.679 46.73,2.701 C42.9729806,4.51194908 40.0995718,7.75449451 38.7536428,11.7020516 C37.4077139,15.6496086 37.701799,19.9721186 39.57,23.701 C41.08,26.641 43.57,29.121 45.91,31.581 C53.03,39.141 60.38,46.491 67.45,54.111 C72.4175495,59.4492221 74.4526451,66.8835066 72.8965704,74.0075359 C71.3404956,81.1315653 66.390952,87.0402215 59.65,89.821 C59.4938176,89.83842 59.3361824,89.83842 59.18,89.821 L59.18,54.591 C46.6388051,61.0478363 35.3944735,69.759905 26.01,80.291 C11.32,96.671 2.64,117.141 0.01,132.071 L23.96,119.821 C31.96,115.771 39.86,111.821 48.14,107.581 L48.14,175.921 L59.14,187.131 L59.14,101.831 C59.14,101.831 59.39,101.711 60.22,101.261 C63.5480598,99.6738911 66.7772674,97.8873078 69.89,95.911 C77.7130888,90.4306687 82.7479457,81.8029342 83.6709542,72.295947 C84.5939627,62.7889599 81.3127806,53.3538429 74.69,46.471 C66.49,37.891 58.24,29.351 50.05,20.761 C47.67,18.261 47.72,15.101 50.05,12.881 C52.38,10.661 55.56,10.881 57.96,13.331 L61.38,16.781 C64.1,19.681 66.79,22.611 69.53,25.481 C76.4547149,32.7389629 83.3947303,39.9823123 90.35,47.211 C90.7,47.571 91.12,47.871 91.5,48.211 L91.93,47.951 C91.8351945,47.4695902 91.6876376,47.0000911 91.49,46.551 Z M48.11,94.931 C47.9883217,95.5022568 47.6230065,95.9917791 47.11,96.271 C42.72,98.601 38.29,100.871 33.87,103.141 L17.76,111.401 C24.771203,96.7435071 35.1132853,83.9289138 47.96,73.981 C48.08,74.221 48.16,74.301 48.16,74.381 C48.15,81.231 48.17,88.081 48.11,94.931 Z" id="Shape"></path>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 1.9 KiB

After

Width:  |  Height:  |  Size: 2.2 KiB

+6 -3
View File
@@ -1,10 +1,14 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Minio Browser</title>
<title>MinIO Browser</title>
<link rel="icon" type="image/png" sizes="32x32" href="/minio/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="/minio/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="/minio/favicon-16x16.png">
<link rel="stylesheet" href="/minio/loader.css" type="text/css">
</head>
@@ -23,7 +27,6 @@
You are using Internet Explorer version 12.0 or lower. Due to security issues and lack of support for Web Standards it is highly recommended that you upgrade to a modern browser
<ul>
<li>
<a href="http://www.google.com/chrome/">
+1 -1
View File
@@ -38,7 +38,7 @@ export const SideBar = ({ sidebarOpen, clickOutside }) => {
>
<div className="fes-header clearfix hidden-sm hidden-xs">
<img src={logo} alt="" />
<h2>Minio Browser</h2>
<h2>MinIO Browser</h2>
</div>
<div className="fes-list">
{web.LoggedIn() && <BucketSearch />}
+2
View File
@@ -170,10 +170,12 @@ select.form-control {
position: absolute;
left: 2px;
top: 8px;
color: rgba(255, 255, 255, 0.5);
}
.ig-text {
padding-left: 25px;
.placeholder(rgba(255, 255, 255, 0.5))
}
}
+3 -2
View File
@@ -1,7 +1,7 @@
.login {
height: 100vh;
min-height: 500px;
background: @dark-gray;
background: #002a37;
text-align: center;
&:before {
@@ -99,6 +99,7 @@
Chrome autofill fix
-------------------------------*/
input:-webkit-autofill {
-webkit-box-shadow:0 0 0 50px @dark-gray inset !important;
-webkit-box-shadow:0 0 0 50px #002a37 inset !important;
-webkit-text-fill-color: @white !important;
caret-color: white;
}
+9
View File
@@ -50,3 +50,12 @@
-webkit-animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
animation: zoomIn 250ms, spin 700ms 250ms infinite linear;
}
/*----------------------------------------
Placeholder
-----------------------------------------*/
.placeholder(@color) {
::-webkit-input-placeholder { color: @color }
::-moz-placeholder { color: @color }
:-ms-input-placeholder { color: @color }
}
+3 -3
View File
@@ -32,7 +32,7 @@
}
.modal-content {
background-color: @dark-gray;
background-color: #00303f;
}
}
@@ -139,8 +139,8 @@
top: 0;
left: 0;
position: absolute;
border-radius: 3px 0 0px 3px;
background-color: #23282C;
border-radius: 3px 0 0 3px;
background-color: #022631;
}
}
}
+9 -8
View File
@@ -3,7 +3,7 @@
----------------------------*/
.fe-sidebar {
width: @fe-sidebar-width;
background-color: @dark-gray;
background-color: @fe-sidebar-bg;
position: fixed;
height: 100%;
overflow: hidden;
@@ -27,7 +27,7 @@
}
a {
color: rgba(255, 255, 255, 0.58);
color: rgba(255, 255, 255, 0.75);
&:hover {
color: @white;
@@ -47,12 +47,14 @@
}
h2 {
margin: 13px 0 0 10px;
margin: 6px 0 0 13px;
font-weight: normal;
color: @white;
font-size: 20px;
}
img {
width: 32px;
width: 20px;
}
}
@@ -95,7 +97,7 @@
}
&.active {
background-color: #282e32;
background-color: rgba(0,0,0,0.1);
& > a {
color: @white;
@@ -103,7 +105,7 @@
}
&:not(.active):hover {
background-color: rgba(0, 0, 0, 0.1);
background-color: rgba(0, 0, 0, 0.05);
& > a {
color: @white;
@@ -192,8 +194,7 @@
left: 0;
bottom: 0;
z-index: 1;
background: @dark-gray;
color: rgba(255, 255, 255, 0.4);
background-color: rgba(0,0,0,0.1);
font-size: 15px;
font-weight: 400;
width: @fe-sidebar-width;
+2
View File
@@ -14,9 +14,11 @@
File Explorer
----------------------------*/
@fe-sidebar-width : 320px;
@fe-sidebar-bg : #00303f;
@text-muted-color : #BDBDBD;
@text-strong-color : #333;
/*--------------------------
Colors
----------------------------*/
+83 -55
View File
File diff suppressed because one or more lines are too long
+3 -1
View File
@@ -93,11 +93,13 @@ var exports = {
plugins: [
new CopyWebpackPlugin([
{from: 'app/css/loader.css'},
{from: 'app/img/favicon.ico'},
{from: 'app/img/browsers/chrome.png'},
{from: 'app/img/browsers/firefox.png'},
{from: 'app/img/browsers/safari.png'},
{from: 'app/img/logo.svg'},
{from: 'app/img/favicon/favicon-16x16.png'},
{from: 'app/img/favicon/favicon-32x32.png'},
{from: 'app/img/favicon/favicon-96x96.png'},
{from: 'app/index.html'}
]),
new webpack.ContextReplacementPlugin(/moment[\\\/]locale$/, /^\.\/(en)$/),
+3 -1
View File
@@ -69,11 +69,13 @@ var exports = {
plugins: [
new CopyWebpackPlugin([
{from: 'app/css/loader.css'},
{from: 'app/img/favicon.ico'},
{from: 'app/img/browsers/chrome.png'},
{from: 'app/img/browsers/firefox.png'},
{from: 'app/img/browsers/safari.png'},
{from: 'app/img/logo.svg'},
{from: 'app/img/favicon/favicon-16x16.png'},
{from: 'app/img/favicon/favicon-32x32.png'},
{from: 'app/img/favicon/favicon-96x96.png'},
{from: 'app/index.html'}
]),
new webpack.ContextReplacementPlugin(/moment[\\\/]locale$/, /^\.\/(en)$/),
-2
View File
@@ -45,8 +45,6 @@ export MINT_RUN_SECURITY_DIR="$MINT_ROOT_DIR/run/security"
export MINT_MODE="full"
export WGET="wget --quiet --no-check-certificate"
go get github.com/go-ini/ini
./create-data-files.sh
./preinstall.sh
+2 -8
View File
@@ -1,12 +1,6 @@
#!/usr/bin/env bash
set -e
echo "" > coverage.txt
for d in $(go list ./... | grep -v browser); do
CGO_ENABLED=0 go test -v -coverprofile=profile.out -covermode=atomic "$d"
if [ -f profile.out ]; then
cat profile.out >> coverage.txt
rm profile.out
fi
done
CGO_ENABLED=0 go test -v -coverprofile=coverage.txt -covermode=atomic ./...
+4 -5
View File
@@ -319,11 +319,10 @@ function __init__()
mkdir -p "$MINIO_CONFIG_DIR"
mkdir -p "$MINT_DATA_DIR"
if ! go get -u github.com/minio/mc; then
echo "failed to download https://github.com/minio/mc"
exit 1
fi
/bin/cp -a "$(go env GOPATH)"/bin/mc "$WORK_DIR/mc"
git clone https://github.com/minio/mc
cd mc
go build
/bin/cp -a ./mc "$WORK_DIR/mc"
chmod a+x "$WORK_DIR/mc"
+17 -24
View File
@@ -177,11 +177,12 @@ func (a adminAPIHandlers) ServiceStopNRestartHandler(w http.ResponseWriter, r *h
// ServerProperties holds some server information such as, version, region
// uptime, etc..
type ServerProperties struct {
Uptime time.Duration `json:"uptime"`
Version string `json:"version"`
CommitID string `json:"commitID"`
Region string `json:"region"`
SQSARN []string `json:"sqsARN"`
Uptime time.Duration `json:"uptime"`
Version string `json:"version"`
CommitID string `json:"commitID"`
DeploymentID string `json:"deploymentID"`
Region string `json:"region"`
SQSARN []string `json:"sqsARN"`
}
// ServerConnStats holds transferred bytes from/to the server
@@ -256,11 +257,12 @@ func (a adminAPIHandlers) ServerInfoHandler(w http.ResponseWriter, r *http.Reque
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
DeploymentID: globalDeploymentID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
},
},
})
@@ -319,9 +321,8 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
}
vars := mux.Vars(r)
perfType := vars["perfType"]
if perfType == "drive" {
switch perfType := vars["perfType"]; perfType {
case "drive":
info := objectAPI.StorageInfo(ctx)
if !(info.Backend.Type == BackendFS || info.Backend.Type == BackendErasure) {
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrMethodNotAllowed), r.URL)
@@ -344,7 +345,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
// Reply with performance information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
} else if perfType == "cpu" {
case "cpu":
// Get CPU load details from local server's cpu(s)
cpu := localEndpointsCPULoad(globalEndpoints)
// Notify all other Minio peers to report cpu load numbers
@@ -361,7 +362,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
// Reply with cpu load information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
} else if perfType == "mem" {
case "mem":
// Get mem usage details from local server(s)
m := localEndpointsMemUsage(globalEndpoints)
// Notify all other Minio peers to report mem usage numbers
@@ -378,7 +379,7 @@ func (a adminAPIHandlers) PerfInfoHandler(w http.ResponseWriter, r *http.Request
// Reply with mem usage information (across nodes in a
// distributed setup) as json.
writeSuccessResponseJSON(w, jsonBytes)
} else {
default:
writeErrorResponseJSON(ctx, w, errorCodes.ToAPIErr(ErrMethodNotAllowed), r.URL)
}
}
@@ -1479,14 +1480,6 @@ func (a adminAPIHandlers) UpdateAdminCredentialsHandler(w http.ResponseWriter,
return
}
// Notify all other Minio peers to update credentials
for _, nerr := range globalNotificationSys.LoadCredentials() {
if nerr.Err != nil {
logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
logger.LogIf(ctx, nerr.Err)
}
}
// Reply to the client before restarting minio server.
writeSuccessResponseHeadersOnly(w)
}
+27 -85
View File
@@ -21,14 +21,12 @@ import (
"encoding/json"
"fmt"
"net/http"
"runtime"
"strings"
"sync"
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/madmin"
"github.com/minio/minio/pkg/sync/errgroup"
)
// healStatusSummary - overall short summary of a healing sequence
@@ -573,51 +571,22 @@ func (h *healSequence) healMinioSysMeta(metaPrefix string) func() error {
// NOTE: Healing on meta is run regardless
// of any bucket being selected, this is to ensure that
// meta are always upto date and correct.
marker := ""
isTruncated := true
for isTruncated {
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
return objectAPI.HealObjects(h.ctx, minioMetaBucket, metaPrefix, func(bucket string, object string) error {
if h.isQuitting() {
return errHealStopSignalled
}
// Lists all objects under `config` prefix.
objectInfos, err := objectAPI.ListObjectsHeal(h.ctx, minioMetaBucket, metaPrefix,
marker, "", 1000)
if err != nil {
return errFnHealFromAPIErr(h.ctx, err)
res, herr := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove, h.settings.ScanMode)
// Object might have been deleted, by the time heal
// was attempted we ignore this object an move on.
if isErrObjectNotFound(herr) {
return nil
}
for index := range objectInfos.Objects {
if h.isQuitting() {
return errHealStopSignalled
}
o := objectInfos.Objects[index]
res, herr := objectAPI.HealObject(h.ctx, o.Bucket, o.Name, h.settings.DryRun, h.settings.Remove)
// Object might have been deleted, by the time heal
// was attempted we ignore this file an move on.
if isErrObjectNotFound(herr) {
continue
}
if herr != nil {
return herr
}
res.Type = madmin.HealItemBucketMetadata
if err = h.pushHealResultItem(res); err != nil {
return err
}
if herr != nil {
return herr
}
isTruncated = objectInfos.IsTruncated
marker = objectInfos.NextMarker
}
return nil
res.Type = madmin.HealItemBucketMetadata
return h.pushHealResultItem(res)
})
}
}
@@ -720,46 +689,9 @@ func (h *healSequence) healBucket(bucket string) error {
return nil
}
entries := runtime.NumCPU()
marker := ""
isTruncated := true
for isTruncated {
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
}
// Heal numCPU * nodes objects at a time.
objectInfos, err := objectAPI.ListObjectsHeal(h.ctx, bucket,
h.objPrefix, marker, "", entries)
if err != nil {
return errFnHealFromAPIErr(h.ctx, err)
}
g := errgroup.WithNErrs(len(objectInfos.Objects))
for index := range objectInfos.Objects {
index := index
g.Go(func() error {
o := objectInfos.Objects[index]
return h.healObject(o.Bucket, o.Name)
}, index)
}
for _, err := range g.Wait() {
if err != nil {
return err
}
}
isTruncated = objectInfos.IsTruncated
marker = objectInfos.NextMarker
if err = objectAPI.HealObjects(h.ctx, bucket,
h.objPrefix, h.healObject); err != nil {
return errFnHealFromAPIErr(h.ctx, err)
}
return nil
}
@@ -770,13 +702,23 @@ func (h *healSequence) healObject(bucket, object string) error {
return errHealStopSignalled
}
if globalHTTPServer != nil {
// Wait at max 1 minute for an inprogress request
// before proceeding to heal
waitCount := 60
// Any requests in progress, delay the heal.
for globalHTTPServer.GetRequestCount() > 2 && waitCount > 0 {
waitCount--
time.Sleep(1 * time.Second)
}
}
// Get current object layer instance.
objectAPI := newObjectLayerFn()
if objectAPI == nil {
return errServerNotInitialized
}
hri, err := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove)
hri, err := objectAPI.HealObject(h.ctx, bucket, object, h.settings.DryRun, h.settings.Remove, h.settings.ScanMode)
if isErrObjectNotFound(err) {
return nil
}
+28 -7
View File
@@ -21,6 +21,7 @@ import (
"encoding/xml"
"fmt"
"net/http"
"strings"
"github.com/Azure/azure-sdk-for-go/storage"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
@@ -179,6 +180,7 @@ const (
// Minio extended errors.
ErrReadQuorum
ErrWriteQuorum
ErrParentIsObject
ErrStorageFull
ErrRequestBodyParse
ErrObjectExistsAsDirectory
@@ -857,6 +859,11 @@ var errorCodes = errorCodeMap{
Description: "Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed.",
HTTPStatusCode: http.StatusInsufficientStorage,
},
ErrParentIsObject: {
Code: "XMinioParentIsObject",
Description: "Object-prefix is already an object, please choose a different object-prefix name.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrRequestBodyParse: {
Code: "XMinioRequestBodyParse",
Description: "The request body failed to parse.",
@@ -1506,8 +1513,6 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrInvalidEncryptionMethod
case crypto.ErrInvalidCustomerAlgorithm:
apiErr = ErrInvalidSSECustomerAlgorithm
case crypto.ErrInvalidCustomerKey:
apiErr = ErrInvalidSSECustomerKey
case crypto.ErrMissingCustomerKey:
apiErr = ErrMissingSSECustomerKey
case crypto.ErrMissingCustomerKeyMD5:
@@ -1562,6 +1567,8 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
apiErr = ErrObjectExistsAsDirectory
case PrefixAccessDenied:
apiErr = ErrAccessDenied
case ParentIsObject:
apiErr = ErrParentIsObject
case BucketNameInvalid:
apiErr = ErrInvalidBucketName
case BucketNotFound:
@@ -1641,11 +1648,25 @@ func toAPIErrorCode(ctx context.Context, err error) (apiErr APIErrorCode) {
case crypto.Error:
apiErr = ErrObjectTampered
default:
apiErr = ErrInternalError
// Make sure to log the errors which we cannot translate
// to a meaningful S3 API errors. This is added to aid in
// debugging unexpected/unhandled errors.
logger.LogIf(ctx, err)
var ie, iw int
// This work-around is to handle the issue golang/go#30648
if _, ferr := fmt.Fscanf(strings.NewReader(err.Error()),
"request declared a Content-Length of %d but only wrote %d bytes",
&ie, &iw); ferr != nil {
apiErr = ErrInternalError
// Make sure to log the errors which we cannot translate
// to a meaningful S3 API errors. This is added to aid in
// debugging unexpected/unhandled errors.
logger.LogIf(ctx, err)
} else if ie > iw {
apiErr = ErrIncompleteBody
} else {
apiErr = ErrInternalError
// Make sure to log the errors which we cannot translate
// to a meaningful S3 API errors. This is added to aid in
// debugging unexpected/unhandled errors.
logger.LogIf(ctx, err)
}
}
return apiErr
+1 -1
View File
@@ -55,7 +55,7 @@ var toAPIErrorTests = []struct {
// SSE-C errors
{err: crypto.ErrInvalidCustomerAlgorithm, errCode: ErrInvalidSSECustomerAlgorithm},
{err: crypto.ErrMissingCustomerKey, errCode: ErrMissingSSECustomerKey},
{err: crypto.ErrInvalidCustomerKey, errCode: ErrInvalidSSECustomerKey},
{err: crypto.ErrInvalidCustomerKey, errCode: ErrAccessDenied},
{err: crypto.ErrMissingCustomerKeyMD5, errCode: ErrMissingSSECustomerKeyMD5},
{err: crypto.ErrCustomerKeyMD5Mismatch, errCode: ErrSSECustomerKeyMD5Mismatch},
{err: errObjectTampered, errCode: ErrObjectTampered},
+7 -3
View File
@@ -84,9 +84,11 @@ func newStreamingBitrotWriter(disk StorageAPI, volume, filePath string, length i
totalFileSize := bitrotSumsTotalSize + length
err := disk.CreateFile(volume, filePath, totalFileSize, r)
if err != nil {
logger.LogIf(context.Background(), err)
r.CloseWithError(err)
reqInfo := (&logger.ReqInfo{}).AppendTags("storageDisk", disk.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
}
r.CloseWithError(err)
close(bw.canClose)
}()
return bw
@@ -125,7 +127,9 @@ func (b *streamingBitrotReader) ReadAt(buf []byte, offset int64) (int, error) {
streamOffset := (offset/b.shardSize)*int64(b.h.Size()) + offset
b.rc, err = b.disk.ReadFileStream(b.volume, b.filePath, streamOffset, b.tillOffset-streamOffset)
if err != nil {
logger.LogIf(context.Background(), err)
reqInfo := (&logger.ReqInfo{}).AppendTags("storageDisk", b.disk.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return 0, err
}
}
+2 -2
View File
@@ -133,7 +133,7 @@ func newBitrotReader(disk StorageAPI, bucket string, filePath string, tillOffset
// Close all the readers.
func closeBitrotReaders(rs []io.ReaderAt) {
for _, r := range rs {
if br, ok := r.(*streamingBitrotReader); ok {
if br, ok := r.(io.Closer); ok {
br.Close()
}
}
@@ -142,7 +142,7 @@ func closeBitrotReaders(rs []io.ReaderAt) {
// Close all the writers.
func closeBitrotWriters(ws []io.Writer) {
for _, w := range ws {
if bw, ok := w.(*streamingBitrotWriter); ok {
if bw, ok := w.(io.Closer); ok {
bw.Close()
}
}
+3 -19
View File
@@ -22,7 +22,6 @@ import (
"encoding/xml"
"fmt"
"io"
"net"
"net/http"
"net/url"
"path"
@@ -226,7 +225,7 @@ func (api objectAPIHandlers) ListBucketsHandler(w http.ResponseWriter, r *http.R
continue
}
bucketsInfo = append(bucketsInfo, BucketInfo{
Name: strings.Trim(dnsRecord.Key, slashSeparator),
Name: dnsRecord.Key,
Created: dnsRecord.CreationDate,
})
bucketSet.Add(dnsRecord.Key)
@@ -371,13 +370,6 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
// Write success response.
writeSuccessResponseXML(w, encodedSuccessResponse)
// Get host and port from Request.RemoteAddr failing which
// fill them with empty strings.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify deleted event for objects.
for _, dobj := range deletedObjects {
sendEvent(eventArgs{
@@ -389,8 +381,7 @@ func (api objectAPIHandlers) DeleteMultipleObjectsHandler(w http.ResponseWriter,
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
}
@@ -681,12 +672,6 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
w.Header().Set("ETag", `"`+objInfo.ETag+`"`)
w.Header().Set("Location", location)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object created event.
defer sendEvent(eventArgs{
EventName: event.ObjectCreatedPost,
@@ -695,8 +680,7 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
if successRedirect != "" {
+2 -2
View File
@@ -332,7 +332,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
if isObjectDir(object, data.Size()) {
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
return ObjectInfo{}, toObjectErr(err, bucket, object)
@@ -350,7 +350,7 @@ func (cfs *cacheFSObjects) PutObject(ctx context.Context, bucket string, object
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
// Validate input data size and it can never be less than zero.
+2 -2
View File
@@ -423,7 +423,7 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
if delimiter == slashSeparator {
recursive = false
}
walkResultCh, endWalkCh := c.listPool.Release(listParams{bucket, recursive, marker, prefix, false})
walkResultCh, endWalkCh := c.listPool.Release(listParams{bucket, recursive, marker, prefix})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := func(bucket, object string) bool {
@@ -495,7 +495,7 @@ func (c cacheObjects) listCacheObjects(ctx context.Context, bucket, prefix, mark
}
}
params := listParams{bucket, recursive, nextMarker, prefix, false}
params := listParams{bucket, recursive, nextMarker, prefix}
if !eof {
c.listPool.Set(params, walkResultCh, endWalkCh)
}
+1 -1
View File
@@ -487,7 +487,7 @@ func (fs *FSObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, pathutil.Dir(object)) {
return oi, toObjectErr(errFileAccessDenied, bucket, object)
return oi, toObjectErr(errFileParentIsFile, bucket, object)
}
if _, err := fs.statBucketDir(ctx, bucket); err != nil {
+8 -11
View File
@@ -839,8 +839,7 @@ func (fs *FSObjects) putObject(ctx context.Context, bucket string, object string
if isObjectDir(object, data.Size()) {
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
logger.LogIf(ctx, errFileAccessDenied)
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
if err = mkdirAll(pathJoin(fs.fsPath, bucket, object), 0777); err != nil {
logger.LogIf(ctx, err)
@@ -859,8 +858,7 @@ func (fs *FSObjects) putObject(ctx context.Context, bucket string, object string
// Check if an object is present as one of the parent dir.
if fs.parentDirIsObject(ctx, bucket, path.Dir(object)) {
logger.LogIf(ctx, errFileAccessDenied)
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
// Validate input data size and it can never be less than zero.
@@ -1148,8 +1146,7 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
return fs.getObjectInfo(ctx, bucket, entry)
}
heal := false // true only for xl.ListObjectsHeal()
walkResultCh, endWalkCh := fs.listPool.Release(listParams{bucket, recursive, marker, prefix, heal})
walkResultCh, endWalkCh := fs.listPool.Release(listParams{bucket, recursive, marker, prefix})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := func(bucket, object string) bool {
@@ -1203,7 +1200,7 @@ func (fs *FSObjects) ListObjects(ctx context.Context, bucket, prefix, marker, de
}
// Save list routine for the next marker if we haven't reached EOF.
params := listParams{bucket, recursive, nextMarker, prefix, heal}
params := listParams{bucket, recursive, nextMarker, prefix}
if !eof {
fs.listPool.Set(params, walkResultCh, endWalkCh)
}
@@ -1241,7 +1238,7 @@ func (fs *FSObjects) HealFormat(ctx context.Context, dryRun bool) (madmin.HealRe
}
// HealObject - no-op for fs. Valid only for XL.
func (fs *FSObjects) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool) (
func (fs *FSObjects) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (
res madmin.HealResultItem, err error) {
logger.LogIf(ctx, NotImplemented{})
return res, NotImplemented{}
@@ -1254,10 +1251,10 @@ func (fs *FSObjects) HealBucket(ctx context.Context, bucket string, dryRun, remo
return madmin.HealResultItem{}, NotImplemented{}
}
// ListObjectsHeal - list all objects to be healed. Valid only for XL
func (fs *FSObjects) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
// HealObjects - no-op for fs. Valid only for XL.
func (fs *FSObjects) HealObjects(ctx context.Context, bucket, prefix string, fn func(string, string) error) (e error) {
logger.LogIf(ctx, NotImplemented{})
return loi, NotImplemented{}
return NotImplemented{}
}
// ListBucketsHeal - list all buckets to be healed. Valid only for XL
+10 -8
View File
@@ -22,6 +22,8 @@ import (
"os"
"path/filepath"
"testing"
"github.com/minio/minio/pkg/madmin"
)
// Tests for if parent directory is object
@@ -228,8 +230,8 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, backend corruption occurred")
}
if nerr, ok := err.(PrefixAccessDenied); !ok {
t.Fatalf("Expected PrefixAccessDenied, got %#v", err)
if nerr, ok := err.(ParentIsObject); !ok {
t.Fatalf("Expected ParentIsObject, got %#v", err)
} else {
if nerr.Bucket != "bucket" {
t.Fatalf("Expected 'bucket', got %s", nerr.Bucket)
@@ -243,8 +245,8 @@ func TestFSPutObject(t *testing.T) {
if err == nil {
t.Fatal("Unexpected should fail here, backned corruption occurred")
}
if nerr, ok := err.(PrefixAccessDenied); !ok {
t.Fatalf("Expected PrefixAccessDenied, got %#v", err)
if nerr, ok := err.(ParentIsObject); !ok {
t.Fatalf("Expected ParentIsObject, got %#v", err)
} else {
if nerr.Bucket != "bucket" {
t.Fatalf("Expected 'bucket', got %s", nerr.Bucket)
@@ -390,19 +392,19 @@ func TestFSHealObject(t *testing.T) {
defer os.RemoveAll(disk)
obj := initFSObjects(disk, t)
_, err := obj.HealObject(context.Background(), "bucket", "object", false, false)
_, err := obj.HealObject(context.Background(), "bucket", "object", false, false, madmin.HealDeepScan)
if err == nil || !isSameType(err, NotImplemented{}) {
t.Fatalf("Heal Object should return NotImplemented error ")
}
}
// TestFSListObjectHeal - tests for fs ListObjectHeals
func TestFSListObjectsHeal(t *testing.T) {
// TestFSHealObjects - tests for fs HealObjects to return not implemented.
func TestFSHealObjects(t *testing.T) {
disk := filepath.Join(globalTestTmpDir, "minio-"+nextSuffix())
defer os.RemoveAll(disk)
obj := initFSObjects(disk, t)
_, err := obj.ListObjectsHeal(context.Background(), "bucket", "prefix", "marker", "delimiter", 1000)
err := obj.HealObjects(context.Background(), "bucket", "prefix", nil)
if err == nil || !isSameType(err, NotImplemented{}) {
t.Fatalf("Heal Object should return NotImplemented error ")
}
+4 -4
View File
@@ -102,7 +102,7 @@ func (a GatewayUnsupported) ListBucketsHeal(ctx context.Context) (buckets []Buck
}
// HealObject - Not implemented stub
func (a GatewayUnsupported) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool) (h madmin.HealResultItem, e error) {
func (a GatewayUnsupported) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (h madmin.HealResultItem, e error) {
return h, NotImplemented{}
}
@@ -111,9 +111,9 @@ func (a GatewayUnsupported) ListObjectsV2(ctx context.Context, bucket, prefix, c
return result, NotImplemented{}
}
// ListObjectsHeal - Not implemented stub
func (a GatewayUnsupported) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
return loi, NotImplemented{}
// HealObjects - Not implemented stub
func (a GatewayUnsupported) HealObjects(ctx context.Context, bucket, prefix string, fn func(string, string) error) (e error) {
return NotImplemented{}
}
// CopyObject copies a blob from source container to destination container.
+24 -1
View File
@@ -564,12 +564,35 @@ func (a *azureObjects) ListObjects(ctx context.Context, bucket, prefix, marker,
// skip all the entries till we reach the marker.
continue
}
// Populate correct ETag's if possible, this code primarily exists
// because AWS S3 indicates that
//
// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
//
// Objects created by the PUT Object, POST Object, or Copy operation,
// or through the AWS Management Console, and are encrypted by SSE-S3
// or plaintext, have ETags that are an MD5 digest of their object data.
//
// Some applications depend on this behavior refer https://github.com/minio/minio/issues/6550
// So we handle it here and make this consistent.
etag := minio.ToS3ETag(blob.Properties.Etag)
switch {
case blob.Properties.ContentMD5 != "":
b, err := base64.StdEncoding.DecodeString(blob.Properties.ContentMD5)
if err == nil {
etag = hex.EncodeToString(b)
}
case blob.Metadata["md5sum"] != "":
etag = blob.Metadata["md5sum"]
delete(blob.Metadata, "md5sum")
}
objects = append(objects, minio.ObjectInfo{
Bucket: bucket,
Name: blob.Name,
ModTime: time.Time(blob.Properties.LastModified),
Size: blob.Properties.ContentLength,
ETag: minio.ToS3ETag(blob.Properties.Etag),
ETag: etag,
ContentType: blob.Properties.ContentType,
ContentEncoding: blob.Properties.ContentEncoding,
})
+1 -1
View File
@@ -182,7 +182,7 @@ func appendS3MetaToOSSOptions(ctx context.Context, opts []oss.Option, s3Metadata
case k == "X-Amz-Acl":
// Valid values: public-read, private, and public-read-write
opts = append(opts, oss.ObjectACL(oss.ACLType(v)))
case k == "X-Amz-Server-Side​-Encryption":
case k == "X-Amz-Server-Side-Encryption":
opts = append(opts, oss.ServerSideEncryption(v))
case k == "X-Amz-Copy-Source-If-Match":
opts = append(opts, oss.CopySourceIfMatch(v))
+1 -2
View File
@@ -57,7 +57,6 @@ const (
globalMinioDefaultStorageClass = "STANDARD"
globalWindowsOSName = "windows"
globalNetBSDOSName = "netbsd"
globalSolarisOSName = "solaris"
globalMinioModeFS = "mode-server-fs"
globalMinioModeXL = "mode-server-xl"
globalMinioModeDistXL = "mode-server-distributed-xl"
@@ -79,7 +78,7 @@ const (
globalMaxSkewTime = 15 * time.Minute // 15 minutes skew allowed.
// GlobalMultipartExpiry - Expiry duration after which the multipart uploads are deemed stale.
GlobalMultipartExpiry = time.Hour * 24 * 14 // 2 weeks.
GlobalMultipartExpiry = time.Hour * 24 * 3 // 3 days.
// GlobalMultipartCleanupInterval - Cleanup interval when the stale multipart cleanup is initiated.
GlobalMultipartCleanupInterval = time.Hour * 24 // 24 hrs.
+13 -2
View File
@@ -25,6 +25,8 @@ import (
"time"
humanize "github.com/dustin/go-humanize"
"golang.org/x/net/http2"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/pkg/certs"
)
@@ -46,17 +48,22 @@ const (
// DefaultMaxHeaderBytes - default maximum HTTP header size in bytes.
DefaultMaxHeaderBytes = 1 * humanize.MiByte
// DefaultHTTP2MaxConcurrentStreams - default value for HTTP 2.0 maximum concurrent streams allowed.
DefaultHTTP2MaxConcurrentStreams = 1024
)
// Server - extended http.Server supports multiple addresses to serve and enhanced connection handling.
type Server struct {
http.Server
Addrs []string // addresses on which the server listens for new connection.
ReadTimeout time.Duration // timeout used for net.Conn.Read() deadlines.
WriteTimeout time.Duration // timeout used for net.Conn.Write() deadlines.
ShutdownTimeout time.Duration // timeout used for graceful server shutdown.
TCPKeepAliveTimeout time.Duration // timeout used for underneath TCP connection.
UpdateBytesReadFunc func(int) // function to be called to update bytes read in bufConn.
UpdateBytesWrittenFunc func(int) // function to be called to update bytes written in bufConn.
listenerMutex *sync.Mutex // to guard 'listener' field.
listenerMutex sync.Mutex // to guard 'listener' field.
listener *httpListener // HTTP listener for all 'Addrs' field.
inShutdown uint32 // indicates whether the server is in shutdown or not
requestCount int32 // counter holds no. of request in progress.
@@ -120,6 +127,11 @@ func (srv *Server) Start() (err error) {
// Start servicing with listener.
if tlsConfig != nil {
if err = http2.ConfigureServer(&srv.Server, &http2.Server{
MaxConcurrentStreams: DefaultHTTP2MaxConcurrentStreams,
}); err != nil {
return err
}
return srv.Server.Serve(tls.NewListener(listener, tlsConfig))
}
return srv.Server.Serve(listener)
@@ -202,7 +214,6 @@ func NewServer(addrs []string, handler http.Handler, getCert certs.GetCertificat
Addrs: addrs,
ShutdownTimeout: DefaultShutdownTimeout,
TCPKeepAliveTimeout: DefaultTCPKeepAliveTimeout,
listenerMutex: &sync.Mutex{},
}
httpServer.Handler = handler
httpServer.TLSConfig = tlsConfig
-4
View File
@@ -77,10 +77,6 @@ func TestNewServer(t *testing.T) {
t.Fatalf("Case %v: server.TCPKeepAliveTimeout: expected: %v, got: %v", (i + 1), DefaultTCPKeepAliveTimeout, server.TCPKeepAliveTimeout)
}
if server.listenerMutex == nil {
t.Fatalf("Case %v: server.listenerMutex: expected: <non-nil>, got: <nil>", (i + 1))
}
if server.ReadTimeout != DefaultReadTimeout {
t.Fatalf("Case %v: server.ReadTimeout: expected: %v, got: %v", (i + 1), DefaultReadTimeout, server.ReadTimeout)
}
+1 -1
View File
@@ -343,7 +343,7 @@ func logIf(ctx context.Context, err error) {
entry.API.Args.Bucket = hashString(entry.API.Args.Bucket)
entry.API.Args.Object = hashString(entry.API.Args.Object)
entry.RemoteHost = hashString(entry.RemoteHost)
entry.Message = reflect.TypeOf(err).String()
entry.Trace.Message = reflect.TypeOf(err).String()
entry.Trace.Variables = make(map[string]string)
}
+139 -117
View File
@@ -43,7 +43,7 @@ type NotificationSys struct {
targetList *event.TargetList
bucketRulesMap map[string]event.RulesMap
bucketRemoteTargetRulesMap map[string]map[event.TargetID]event.RulesMap
peerRPCClientMap map[xnet.Host]*PeerRPCClient
peerClients []*peerRESTClient
}
// GetARNList - returns available ARNs.
@@ -63,11 +63,6 @@ func (sys *NotificationSys) GetARNList() []string {
return arns
}
// GetPeerRPCClient - returns PeerRPCClient of addr.
func (sys *NotificationSys) GetPeerRPCClient(addr xnet.Host) *PeerRPCClient {
return sys.peerRPCClientMap[addr]
}
// NotificationPeerErr returns error associated for a remote peer.
type NotificationPeerErr struct {
Host xnet.Host // Remote host on which the rpc call was initiated
@@ -78,15 +73,15 @@ type NotificationPeerErr struct {
func (sys *NotificationSys) DeleteBucket(ctx context.Context, bucketName string) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.DeleteBucket(bucketName); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -146,52 +141,45 @@ func (g *NotificationGroup) Go(ctx context.Context, f func() error, index int, a
}()
}
// ReloadFormat - calls ReloadFormat RPC call on all peers.
// ReloadFormat - calls ReloadFormat REST call on all peers.
func (sys *NotificationSys) ReloadFormat(dryRun bool) []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), func() error {
return client.ReloadFormat(dryRun)
}, idx, addr)
idx++
}, idx, *client.host)
}
return ng.Wait()
}
// LoadUsers - calls LoadUsers RPC call on all peers.
func (sys *NotificationSys) LoadUsers() []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng.Go(context.Background(), client.LoadUsers, idx, addr)
idx++
}
return ng.Wait()
}
// LoadCredentials - calls LoadCredentials RPC call on all peers.
func (sys *NotificationSys) LoadCredentials() []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng.Go(context.Background(), client.LoadCredentials, idx, addr)
idx++
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), client.LoadUsers, idx, *client.host)
}
return ng.Wait()
}
// StartProfiling - start profiling on remote peers, by initiating a remote RPC.
func (sys *NotificationSys) StartProfiling(profiler string) []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), func() error {
return client.StartProfiling(profiler)
}, idx, addr)
idx++
}, idx, *client.host)
}
return ng.Wait()
}
@@ -205,10 +193,13 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
zipWriter := zip.NewWriter(writer)
defer zipWriter.Close()
for addr, client := range sys.peerRPCClientMap {
data, err := client.DownloadProfilingData()
for _, client := range sys.peerClients {
if client == nil {
continue
}
data, err := client.DownloadProfileData()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, err)
continue
@@ -218,7 +209,7 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
// Send profiling data to zip as file
header, zerr := zip.FileInfoHeader(dummyFileInfo{
name: fmt.Sprintf("profiling-%s.pprof", addr),
name: fmt.Sprintf("profiling-%s.pprof", client.host.String()),
size: int64(len(data)),
mode: 0600,
modTime: UTCNow(),
@@ -226,20 +217,20 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
sys: nil,
})
if zerr != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, zerr)
continue
}
zwriter, zerr := zipWriter.CreateHeader(header)
if zerr != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, zerr)
continue
}
if _, err = io.Copy(zwriter, bytes.NewBuffer(data)); err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, err)
continue
@@ -289,45 +280,48 @@ func (sys *NotificationSys) DownloadProfilingData(ctx context.Context, writer io
// SignalService - calls signal service RPC call on all peers.
func (sys *NotificationSys) SignalService(sig serviceSignal) []NotificationPeerErr {
var idx = 0
ng := WithNPeers(len(sys.peerRPCClientMap))
for addr, client := range sys.peerRPCClientMap {
ng := WithNPeers(len(sys.peerClients))
for idx, client := range sys.peerClients {
if client == nil {
continue
}
client := client
ng.Go(context.Background(), func() error {
return client.SignalService(sig)
}, idx, addr)
idx++
}, idx, *client.host)
}
return ng.Wait()
}
// ServerInfo - calls ServerInfo RPC call on all peers.
func (sys *NotificationSys) ServerInfo(ctx context.Context) []ServerInfo {
var idx = 0
serverInfo := make([]ServerInfo, len(sys.peerRPCClientMap))
serverInfo := make([]ServerInfo, len(sys.peerClients))
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for index, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(idx int, addr xnet.Host, client *PeerRPCClient) {
go func(idx int, client *peerRESTClient) {
defer wg.Done()
// Try to fetch serverInfo remotely in three attempts.
for i := 0; i < 3; i++ {
info, err := client.ServerInfo()
if err == nil {
serverInfo[idx] = ServerInfo{
Addr: addr.String(),
Addr: client.host.String(),
Data: &info,
}
return
}
serverInfo[idx] = ServerInfo{
Addr: addr.String(),
Addr: client.host.String(),
Data: &info,
Error: err.Error(),
}
// Last iteration log the error.
if i == 2 {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogIf(ctx, err)
}
@@ -336,8 +330,7 @@ func (sys *NotificationSys) ServerInfo(ctx context.Context) []ServerInfo {
time.Sleep(1 * time.Second)
}
}
}(idx, addr, client)
idx++
}(index, client)
}
wg.Wait()
return serverInfo
@@ -345,19 +338,22 @@ func (sys *NotificationSys) ServerInfo(ctx context.Context) []ServerInfo {
// GetLocks - makes GetLocks RPC call on all peers.
func (sys *NotificationSys) GetLocks(ctx context.Context) []*PeerLocks {
var idx = 0
locksResp := make([]*PeerLocks, len(sys.peerRPCClientMap))
locksResp := make([]*PeerLocks, len(sys.peerClients))
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for index, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(idx int, addr xnet.Host, client *PeerRPCClient) {
go func(idx int, client *peerRESTClient) {
defer wg.Done()
// Try to fetch serverInfo remotely in three attempts.
for i := 0; i < 3; i++ {
serverLocksResp, err := client.GetLocks()
if err == nil {
locksResp[idx] = &PeerLocks{
Addr: addr.String(),
Addr: client.host.String(),
Locks: serverLocksResp,
}
return
@@ -365,17 +361,16 @@ func (sys *NotificationSys) GetLocks(ctx context.Context) []*PeerLocks {
// Last iteration log the error.
if i == 2 {
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("peerAddress", client.host.String())
ctx := logger.SetReqInfo(ctx, reqInfo)
logger.LogOnceIf(ctx, err, addr.String())
logger.LogOnceIf(ctx, err, client.host.String())
}
// Wait for one second and no need wait after last attempt.
if i < 2 {
time.Sleep(1 * time.Second)
}
}
}(idx, addr, client)
idx++
}(index, client)
}
wg.Wait()
return locksResp
@@ -385,15 +380,18 @@ func (sys *NotificationSys) GetLocks(ctx context.Context) []*PeerLocks {
func (sys *NotificationSys) SetBucketPolicy(ctx context.Context, bucketName string, bucketPolicy *policy.Policy) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.SetBucketPolicy(bucketName, bucketPolicy); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -403,15 +401,18 @@ func (sys *NotificationSys) SetBucketPolicy(ctx context.Context, bucketName stri
func (sys *NotificationSys) RemoveBucketPolicy(ctx context.Context, bucketName string) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.RemoveBucketPolicy(bucketName); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -421,15 +422,18 @@ func (sys *NotificationSys) RemoveBucketPolicy(ctx context.Context, bucketName s
func (sys *NotificationSys) PutBucketNotification(ctx context.Context, bucketName string, rulesMap event.RulesMap) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, rulesMap event.RulesMap) {
go func(client *peerRESTClient, rulesMap event.RulesMap) {
defer wg.Done()
if err := client.PutBucketNotification(bucketName, rulesMap); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client, rulesMap.Clone())
}(client, rulesMap.Clone())
}
wg.Wait()
}()
@@ -440,15 +444,18 @@ func (sys *NotificationSys) ListenBucketNotification(ctx context.Context, bucket
targetID event.TargetID, localPeer xnet.Host) {
go func() {
var wg sync.WaitGroup
for addr, client := range sys.peerRPCClientMap {
for _, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient) {
go func(client *peerRESTClient) {
defer wg.Done()
if err := client.ListenBucketNotification(bucketName, eventNames, pattern, targetID, localPeer); err != nil {
logger.GetReqInfo(ctx).AppendTags("remotePeer", addr.Name)
logger.GetReqInfo(ctx).AppendTags("remotePeer", client.host.Name)
logger.LogIf(ctx, err)
}
}(addr, client)
}(client)
}
wg.Wait()
}()
@@ -492,7 +499,17 @@ func (sys *NotificationSys) RemoteTargetExist(bucketName string, targetID event.
return ok
}
// initListeners - initializes PeerRPC clients available in listener.json.
// ListenBucketNotificationArgs - listen bucket notification RPC arguments.
type ListenBucketNotificationArgs struct {
AuthArgs `json:"-"`
BucketName string `json:"-"`
EventNames []event.Name `json:"eventNames"`
Pattern string `json:"pattern"`
TargetID event.TargetID `json:"targetId"`
Addr xnet.Host `json:"addr"`
}
// initListeners - initializes PeerREST clients available in listener.json.
func (sys *NotificationSys) initListeners(ctx context.Context, objAPI ObjectLayer, bucketName string) error {
// listener.json is available/applicable only in DistXL mode.
if !globalIsDistXL {
@@ -542,12 +559,12 @@ func (sys *NotificationSys) initListeners(ctx context.Context, objAPI ObjectLaye
continue
}
rpcClient := sys.GetPeerRPCClient(args.Addr)
if rpcClient == nil {
return fmt.Errorf("unable to find PeerRPCClient by address %v in listener.json for bucket %v", args.Addr, bucketName)
client, err := newPeerRESTClient(&args.Addr)
if err != nil {
return fmt.Errorf("unable to find PeerHost by address %v in listener.json for bucket %v", args.Addr, bucketName)
}
exist, err := rpcClient.RemoteTargetExist(bucketName, args.TargetID)
exist, err := client.RemoteTargetExist(bucketName, args.TargetID)
if err != nil {
logger.GetReqInfo(ctx).AppendTags("targetID", args.TargetID.Name)
logger.LogIf(ctx, err)
@@ -558,7 +575,7 @@ func (sys *NotificationSys) initListeners(ctx context.Context, objAPI ObjectLaye
continue
}
target := NewPeerRPCClientTarget(bucketName, args.TargetID, rpcClient)
target := NewPeerRESTClientTarget(bucketName, args.TargetID, client)
rulesMap := event.NewRulesMap(args.EventNames, args.Pattern, target.ID())
if err = sys.AddRemoteTarget(bucketName, target, rulesMap); err != nil {
logger.GetReqInfo(ctx).AppendTags("targetName", target.id.Name)
@@ -721,24 +738,25 @@ func (sys *NotificationSys) Send(args eventArgs) []event.TargetIDErr {
// DrivePerfInfo - Drive speed (read and write) information
func (sys *NotificationSys) DrivePerfInfo() []ServerDrivesPerfInfo {
reply := make([]ServerDrivesPerfInfo, len(sys.peerRPCClientMap))
reply := make([]ServerDrivesPerfInfo, len(sys.peerClients))
var wg sync.WaitGroup
var i int
for addr, client := range sys.peerRPCClientMap {
for i, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, idx int) {
go func(client *peerRESTClient, idx int) {
defer wg.Done()
di, err := client.DrivePerfInfo()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", client.host.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
di.Addr = addr.String()
di.Addr = client.host.String()
di.Error = err.Error()
}
reply[idx] = di
}(addr, client, i)
i++
}(client, i)
}
wg.Wait()
return reply
@@ -746,24 +764,25 @@ func (sys *NotificationSys) DrivePerfInfo() []ServerDrivesPerfInfo {
// MemUsageInfo - Mem utilization information
func (sys *NotificationSys) MemUsageInfo() []ServerMemUsageInfo {
reply := make([]ServerMemUsageInfo, len(sys.peerRPCClientMap))
reply := make([]ServerMemUsageInfo, len(sys.peerClients))
var wg sync.WaitGroup
var i int
for addr, client := range sys.peerRPCClientMap {
for i, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, idx int) {
go func(client *peerRESTClient, idx int) {
defer wg.Done()
memi, err := client.MemUsageInfo()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", client.host.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
memi.Addr = addr.String()
memi.Addr = client.host.String()
memi.Error = err.Error()
}
reply[idx] = memi
}(addr, client, i)
i++
}(client, i)
}
wg.Wait()
return reply
@@ -771,24 +790,25 @@ func (sys *NotificationSys) MemUsageInfo() []ServerMemUsageInfo {
// CPULoadInfo - CPU utilization information
func (sys *NotificationSys) CPULoadInfo() []ServerCPULoadInfo {
reply := make([]ServerCPULoadInfo, len(sys.peerRPCClientMap))
reply := make([]ServerCPULoadInfo, len(sys.peerClients))
var wg sync.WaitGroup
var i int
for addr, client := range sys.peerRPCClientMap {
for i, client := range sys.peerClients {
if client == nil {
continue
}
wg.Add(1)
go func(addr xnet.Host, client *PeerRPCClient, idx int) {
go func(client *peerRESTClient, idx int) {
defer wg.Done()
cpui, err := client.CPULoadInfo()
if err != nil {
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", addr.String())
reqInfo := (&logger.ReqInfo{}).AppendTags("remotePeer", client.host.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
cpui.Addr = addr.String()
cpui.Addr = client.host.String()
cpui.Error = err.Error()
}
reply[idx] = cpui
}(addr, client, i)
i++
}(client, i)
}
wg.Wait()
return reply
@@ -797,14 +817,18 @@ func (sys *NotificationSys) CPULoadInfo() []ServerCPULoadInfo {
// NewNotificationSys - creates new notification system object.
func NewNotificationSys(config *serverConfig, endpoints EndpointList) *NotificationSys {
targetList := getNotificationTargets(config)
peerRPCClientMap := makeRemoteRPCClients(endpoints)
remoteHosts := getRemoteHosts(endpoints)
remoteClients, err := getRestClients(remoteHosts)
if err != nil {
logger.FatalIf(err, "Unable to start notification sub system")
}
// bucketRulesMap/bucketRemoteTargetRulesMap are initialized by NotificationSys.Init()
return &NotificationSys{
targetList: targetList,
bucketRulesMap: make(map[string]event.RulesMap),
bucketRemoteTargetRulesMap: make(map[string]map[event.TargetID]event.RulesMap),
peerRPCClientMap: peerRPCClientMap,
peerClients: remoteClients,
}
}
@@ -815,7 +839,6 @@ type eventArgs struct {
ReqParams map[string]string
RespElements map[string]string
Host string
Port string
UserAgent string
}
@@ -870,7 +893,6 @@ func (args eventArgs) ToEvent() event.Event {
},
Source: event.Source{
Host: args.Host,
Port: args.Port,
UserAgent: args.UserAgent,
},
}
+15
View File
@@ -19,6 +19,7 @@ package cmd
import (
"fmt"
"io"
"path"
)
// Converts underlying storage error. Convenience function written to
@@ -47,6 +48,13 @@ func toObjectErr(err error, params ...string) error {
Object: params[1],
}
}
case errFileParentIsFile:
if len(params) >= 2 {
err = ParentIsObject{
Bucket: params[0],
Object: params[1],
}
}
case errIsNotRegular:
if len(params) >= 2 {
err = ObjectExistsAsDirectory{
@@ -182,6 +190,13 @@ func (e PrefixAccessDenied) Error() string {
return "Prefix access is denied: " + e.Bucket + "/" + e.Object
}
// ParentIsObject object access is denied.
type ParentIsObject GenericError
func (e ParentIsObject) Error() string {
return "Parent is object " + e.Bucket + "/" + path.Dir(e.Object)
}
// BucketExists bucket exists.
type BucketExists GenericError
+2 -2
View File
@@ -88,9 +88,9 @@ type ObjectLayer interface {
ReloadFormat(ctx context.Context, dryRun bool) error
HealFormat(ctx context.Context, dryRun bool) (madmin.HealResultItem, error)
HealBucket(ctx context.Context, bucket string, dryRun, remove bool) (madmin.HealResultItem, error)
HealObject(ctx context.Context, bucket, object string, dryRun, remove bool) (madmin.HealResultItem, error)
HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (madmin.HealResultItem, error)
ListBucketsHeal(ctx context.Context) (buckets []BucketInfo, err error)
ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, error)
HealObjects(ctx context.Context, bucket, prefix string, healObjectFn func(string, string) error) error
// Policy operations
SetBucketPolicy(context.Context, string, *policy.Policy) error
+5 -9
View File
@@ -18,7 +18,6 @@ package cmd
import (
"context"
"net"
"net/http"
"strings"
"time"
@@ -99,13 +98,14 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
}
}
ssec := crypto.SSECopy.IsRequested(r.Header)
shouldDecryptEtag := crypto.SSECopy.IsRequested(r.Header) && !crypto.IsMultiPart(objInfo.UserDefined)
// x-amz-copy-source-if-match : Return the object only if its entity tag (ETag) is the
// same as the one specified; otherwise return a 412 (precondition failed).
ifMatchETagHeader := r.Header.Get("x-amz-copy-source-if-match")
if ifMatchETagHeader != "" {
etag := objInfo.ETag
if ssec {
if shouldDecryptEtag {
etag = encETag[len(encETag)-32:]
}
if objInfo.ETag != "" && !isETagEqual(etag, ifMatchETagHeader) {
@@ -121,7 +121,7 @@ func checkCopyObjectPreconditions(ctx context.Context, w http.ResponseWriter, r
ifNoneMatchETagHeader := r.Header.Get("x-amz-copy-source-if-none-match")
if ifNoneMatchETagHeader != "" {
etag := objInfo.ETag
if ssec {
if shouldDecryptEtag {
etag = encETag[len(encETag)-32:]
}
if objInfo.ETag != "" && isETagEqual(etag, ifNoneMatchETagHeader) {
@@ -253,9 +253,6 @@ func deleteObject(ctx context.Context, obj ObjectLayer, cache CacheObjectLayer,
return err
}
// Get host and port from Request.RemoteAddr.
host, port, _ := net.SplitHostPort(handlers.GetSourceIP(r))
// Notify object deleted event.
sendEvent(eventArgs{
EventName: event.ObjectRemovedDelete,
@@ -265,8 +262,7 @@ func deleteObject(ctx context.Context, obj ObjectLayer, cache CacheObjectLayer,
},
ReqParams: extractReqParams(r),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
return nil
+7 -47
View File
@@ -24,7 +24,6 @@ import (
"encoding/xml"
"io"
goioutil "io/ioutil"
"net"
"net/http"
"net/url"
"sort"
@@ -221,12 +220,6 @@ func (api objectAPIHandlers) SelectObjectContentHandler(w http.ResponseWriter, r
s3Select.Evaluate(w)
s3Select.Close()
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object accessed via a GET request.
sendEvent(eventArgs{
EventName: event.ObjectAccessedGet,
@@ -235,8 +228,7 @@ func (api objectAPIHandlers) SelectObjectContentHandler(w http.ResponseWriter, r
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -402,12 +394,6 @@ func (api objectAPIHandlers) GetObjectHandler(w http.ResponseWriter, r *http.Req
}
}
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object accessed via a GET request.
sendEvent(eventArgs{
EventName: event.ObjectAccessedGet,
@@ -416,8 +402,7 @@ func (api objectAPIHandlers) GetObjectHandler(w http.ResponseWriter, r *http.Req
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -562,11 +547,6 @@ func (api objectAPIHandlers) HeadObjectHandler(w http.ResponseWriter, r *http.Re
w.WriteHeader(http.StatusOK)
}
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object accessed via a HEAD request.
sendEvent(eventArgs{
EventName: event.ObjectAccessedHead,
@@ -575,8 +555,7 @@ func (api objectAPIHandlers) HeadObjectHandler(w http.ResponseWriter, r *http.Re
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -1032,12 +1011,6 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
// Write success response.
writeSuccessResponseXML(w, encodedSuccessResponse)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
if objInfo.IsCompressed() {
objInfo.Size = actualSize
}
@@ -1050,8 +1023,7 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -1303,12 +1275,6 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
writeSuccessResponseHeadersOnly(w)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object created event.
sendEvent(eventArgs{
EventName: event.ObjectCreatedPut,
@@ -1317,8 +1283,7 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -2345,16 +2310,12 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
writeSuccessResponseXML(w, encodedSuccessResponse)
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
if objectAPI.IsEncryptionSupported() {
if crypto.IsEncrypted(objInfo.UserDefined) {
objInfo.Size, _ = objInfo.DecryptedSize()
}
}
// Notify object created event.
sendEvent(eventArgs{
EventName: event.ObjectCreatedCompleteMultipartUpload,
@@ -2363,8 +2324,7 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
+4
View File
@@ -142,6 +142,10 @@ func renameAll(srcFilePath, dstFilePath string) (err error) {
return fmt.Errorf("%s (%s)->(%s)", errCrossDeviceLink, srcFilePath, dstFilePath)
case os.IsNotExist(err):
return errFileNotFound
case os.IsExist(err):
// This is returned only when destination is a directory and we
// are attempting a rename from file to directory.
return errIsNotRegular
default:
return err
}
@@ -18,35 +18,35 @@ package cmd
import "github.com/minio/minio/pkg/event"
// PeerRPCClientTarget - RPCClient is an event.Target which sends event to target of remote peer.
type PeerRPCClientTarget struct {
// PeerRESTClientTarget - RPCClient is an event.Target which sends event to target of remote peer.
type PeerRESTClientTarget struct {
id event.TargetID
remoteTargetID event.TargetID
rpcClient *PeerRPCClient
restClient *peerRESTClient
bucketName string
}
// ID - returns target ID.
func (target *PeerRPCClientTarget) ID() event.TargetID {
func (target *PeerRESTClientTarget) ID() event.TargetID {
return target.id
}
// Send - sends event to remote peer by making RPC call.
func (target *PeerRPCClientTarget) Send(eventData event.Event) error {
return target.rpcClient.SendEvent(target.bucketName, target.id, target.remoteTargetID, eventData)
func (target *PeerRESTClientTarget) Send(eventData event.Event) error {
return target.restClient.SendEvent(target.bucketName, target.id, target.remoteTargetID, eventData)
}
// Close - does nothing and available for interface compatibility.
func (target *PeerRPCClientTarget) Close() error {
func (target *PeerRESTClientTarget) Close() error {
return nil
}
// NewPeerRPCClientTarget - creates RPCClient target with given target ID available in remote peer.
func NewPeerRPCClientTarget(bucketName string, targetID event.TargetID, rpcClient *PeerRPCClient) *PeerRPCClientTarget {
return &PeerRPCClientTarget{
// NewPeerRESTClientTarget - creates RPCClient target with given target ID available in remote peer.
func NewPeerRESTClientTarget(bucketName string, targetID event.TargetID, restClient *peerRESTClient) *PeerRESTClientTarget {
return &PeerRESTClientTarget{
id: event.TargetID{ID: targetID.ID, Name: targetID.Name + "+" + mustGetUUID()},
remoteTargetID: targetID,
bucketName: bucketName,
rpcClient: rpcClient,
restClient: restClient,
}
}
+415
View File
@@ -0,0 +1,415 @@
/*
* Minio Cloud Storage, (C) 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"bytes"
"context"
"crypto/tls"
"encoding/gob"
"io"
"net/url"
"github.com/minio/minio/cmd/http"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/cmd/rest"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
// client to talk to peer Nodes.
type peerRESTClient struct {
host *xnet.Host
restClient *rest.Client
connected bool
}
// Reconnect to a peer rest server.
func (client *peerRESTClient) reConnect() error {
// correct (intelligent) retry logic will be
// implemented in subsequent PRs.
client.connected = true
return nil
}
// Wrapper to restClient.Call to handle network errors, in case of network error the connection is marked disconnected
// permanently. The only way to restore the connection is at the xl-sets layer by xlsets.monitorAndConnectEndpoints()
// after verifying format.json
func (client *peerRESTClient) call(method string, values url.Values, body io.Reader, length int64) (respBody io.ReadCloser, err error) {
if !client.connected {
err := client.reConnect()
logger.LogIf(context.Background(), err)
if err != nil {
return nil, err
}
}
if values == nil {
values = make(url.Values)
}
respBody, err = client.restClient.Call(method, values, body, length)
if err == nil {
return respBody, nil
}
if isNetworkError(err) {
client.connected = false
}
return nil, err
}
// Stringer provides a canonicalized representation of node.
func (client *peerRESTClient) String() string {
return client.host.String()
}
// IsOnline - returns whether RPC client failed to connect or not.
func (client *peerRESTClient) IsOnline() bool {
return client.connected
}
// Close - marks the client as closed.
func (client *peerRESTClient) Close() error {
client.connected = false
client.restClient.Close()
return nil
}
// GetLocksResp stores various info from the client for each lock that is requested.
type GetLocksResp map[string][]lockRequesterInfo
// GetLocks - fetch older locks for a remote node.
func (client *peerRESTClient) GetLocks() (locks GetLocksResp, err error) {
respBody, err := client.call(peerRESTMethodGetLocks, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&locks)
return locks, err
}
// ServerInfo - fetch server information for a remote node.
func (client *peerRESTClient) ServerInfo() (info ServerInfoData, err error) {
respBody, err := client.call(peerRESTMethodServerInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// CPULoadInfo - fetch CPU information for a remote node.
func (client *peerRESTClient) CPULoadInfo() (info ServerCPULoadInfo, err error) {
respBody, err := client.call(peerRESTMethodCPULoadInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// DrivePerfInfo - fetch Drive performance information for a remote node.
func (client *peerRESTClient) DrivePerfInfo() (info ServerDrivesPerfInfo, err error) {
respBody, err := client.call(peerRESTMethodDrivePerfInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// MemUsageInfo - fetch memory usage information for a remote node.
func (client *peerRESTClient) MemUsageInfo() (info ServerMemUsageInfo, err error) {
respBody, err := client.call(peerRESTMethodMemUsageInfo, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&info)
return info, err
}
// StartProfiling - Issues profiling command on the peer node.
func (client *peerRESTClient) StartProfiling(profiler string) error {
values := make(url.Values)
values.Set(peerRESTProfiler, profiler)
respBody, err := client.call(peerRESTMethodStartProfiling, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// DownloadProfileData - download profiled data from a remote node.
func (client *peerRESTClient) DownloadProfileData() (data []byte, err error) {
respBody, err := client.call(peerRESTMethodDownloadProfilingData, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&data)
return data, err
}
// DeleteBucket - Delete notification and policies related to the bucket.
func (client *peerRESTClient) DeleteBucket(bucket string) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
respBody, err := client.call(peerRESTMethodDeleteBucket, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// ReloadFormat - reload format on the peer node.
func (client *peerRESTClient) ReloadFormat(dryRun bool) error {
values := make(url.Values)
if dryRun {
values.Set(peerRESTDryRun, "true")
} else {
values.Set(peerRESTDryRun, "false")
}
respBody, err := client.call(peerRESTMethodReloadFormat, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// ListenBucketNotification - send listent bucket notification to peer nodes.
func (client *peerRESTClient) ListenBucketNotification(bucket string, eventNames []event.Name,
pattern string, targetID event.TargetID, addr xnet.Host) error {
args := listenBucketNotificationReq{
EventNames: eventNames,
Pattern: pattern,
TargetID: targetID,
Addr: addr,
}
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(args)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodBucketNotificationListen, values, &reader, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// SendEvent - calls send event RPC.
func (client *peerRESTClient) SendEvent(bucket string, targetID, remoteTargetID event.TargetID, eventData event.Event) error {
args := sendEventRequest{
TargetID: remoteTargetID,
Event: eventData,
}
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(args)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodSendEvent, values, &reader, -1)
if err != nil {
return err
}
var eventResp sendEventResp
defer http.DrainBody(respBody)
err = gob.NewDecoder(respBody).Decode(&eventResp)
if err != nil || !eventResp.Success {
reqInfo := &logger.ReqInfo{BucketName: bucket}
reqInfo.AppendTags("targetID", targetID.Name)
reqInfo.AppendTags("event", eventData.EventName.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
globalNotificationSys.RemoveRemoteTarget(bucket, targetID)
}
return err
}
// RemoteTargetExist - calls remote target ID exist REST API.
func (client *peerRESTClient) RemoteTargetExist(bucket string, targetID event.TargetID) (bool, error) {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(targetID)
if err != nil {
return false, err
}
respBody, err := client.call(peerRESTMethodTargetExists, values, &reader, -1)
if err != nil {
return false, err
}
defer http.DrainBody(respBody)
var targetExists remoteTargetExistsResp
err = gob.NewDecoder(respBody).Decode(&targetExists)
return targetExists.Exists, err
}
// RemoveBucketPolicy - Remove bucket policy on the peer node.
func (client *peerRESTClient) RemoveBucketPolicy(bucket string) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
respBody, err := client.call(peerRESTMethodBucketPolicyRemove, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// SetBucketPolicy - Set bucket policy on the peer node.
func (client *peerRESTClient) SetBucketPolicy(bucket string, bucketPolicy *policy.Policy) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(bucketPolicy)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodBucketPolicySet, values, &reader, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// PutBucketNotification - Put bucket notification on the peer node.
func (client *peerRESTClient) PutBucketNotification(bucket string, rulesMap event.RulesMap) error {
values := make(url.Values)
values.Set(peerRESTBucket, bucket)
var reader bytes.Buffer
err := gob.NewEncoder(&reader).Encode(&rulesMap)
if err != nil {
return err
}
respBody, err := client.call(peerRESTMethodBucketNotificationPut, values, &reader, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
// LoadUsers - send load users command to peer nodes.
func (client *peerRESTClient) LoadUsers() (err error) {
respBody, err := client.call(peerRESTMethodLoadUsers, nil, nil, -1)
if err != nil {
return
}
defer http.DrainBody(respBody)
return nil
}
// SignalService - sends signal to peer nodes.
func (client *peerRESTClient) SignalService(sig serviceSignal) error {
values := make(url.Values)
values.Set(peerRESTSignal, string(sig))
respBody, err := client.call(peerRESTMethodSignalService, values, nil, -1)
if err != nil {
return err
}
defer http.DrainBody(respBody)
return nil
}
func getRemoteHosts(endpoints EndpointList) []*xnet.Host {
var remoteHosts []*xnet.Host
for _, hostStr := range GetRemotePeers(endpoints) {
host, err := xnet.ParseHost(hostStr)
logger.FatalIf(err, "Unable to parse peer Host")
remoteHosts = append(remoteHosts, host)
}
return remoteHosts
}
func getRestClients(peerHosts []*xnet.Host) ([]*peerRESTClient, error) {
restClients := make([]*peerRESTClient, len(peerHosts))
for i, host := range peerHosts {
client, err := newPeerRESTClient(host)
if err != nil {
logger.LogIf(context.Background(), err)
}
restClients[i] = client
}
return restClients, nil
}
// Returns a peer rest client.
func newPeerRESTClient(peer *xnet.Host) (*peerRESTClient, error) {
scheme := "http"
if globalIsSSL {
scheme = "https"
}
serverURL := &url.URL{
Scheme: scheme,
Host: peer.String(),
Path: peerRESTPath,
}
var tlsConfig *tls.Config
if globalIsSSL {
tlsConfig = &tls.Config{
ServerName: peer.Name,
RootCAs: globalRootCAs,
}
}
restClient, err := rest.NewClient(serverURL, tlsConfig, rest.DefaultRESTTimeout, newAuthToken)
if err != nil {
return &peerRESTClient{host: peer, restClient: restClient, connected: false}, err
}
return &peerRESTClient{host: peer, restClient: restClient, connected: true}, nil
}
+47
View File
@@ -0,0 +1,47 @@
/*
* Minio Cloud Storage, (C) 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
const peerRESTVersion = "v1"
const peerRESTPath = minioReservedBucketPath + "/peer/" + peerRESTVersion
const (
peerRESTMethodServerInfo = "serverinfo"
peerRESTMethodCPULoadInfo = "cpuloadinfo"
peerRESTMethodMemUsageInfo = "memusageinfo"
peerRESTMethodDrivePerfInfo = "driveperfinfo"
peerRESTMethodDeleteBucket = "deletebucket"
peerRESTMethodSignalService = "signalservice"
peerRESTMethodGetLocks = "getlocks"
peerRESTMethodBucketPolicyRemove = "removebucketpolicy"
peerRESTMethodLoadUsers = "loadusers"
peerRESTMethodStartProfiling = "startprofiling"
peerRESTMethodDownloadProfilingData = "downloadprofilingdata"
peerRESTMethodBucketPolicySet = "setbucketpolicy"
peerRESTMethodBucketNotificationPut = "putbucketnotification"
peerRESTMethodBucketNotificationListen = "listenbucketnotification"
peerRESTMethodReloadFormat = "reloadformat"
peerRESTMethodTargetExists = "targetexists"
peerRESTMethodSendEvent = "sendevent"
)
const (
peerRESTBucket = "bucket"
peerRESTSignal = "signal"
peerRESTProfiler = "profiler"
peerRESTDryRun = "dry-run"
)
+592
View File
@@ -0,0 +1,592 @@
/*
* Minio Cloud Storage, (C) 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"encoding/gob"
"errors"
"fmt"
"net/http"
"sort"
"strings"
"time"
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
// To abstract a node over network.
type peerRESTServer struct {
}
func getServerInfo() (*ServerInfoData, error) {
if globalBootTime.IsZero() {
return nil, errServerNotInitialized
}
objLayer := newObjectLayerFn()
if objLayer == nil {
return nil, errServerNotInitialized
}
// Server info data.
return &ServerInfoData{
StorageInfo: objLayer.StorageInfo(context.Background()),
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
DeploymentID: globalDeploymentID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
},
}, nil
}
// uptimes - used to sort uptimes in chronological order.
type uptimes []time.Duration
func (ts uptimes) Len() int {
return len(ts)
}
func (ts uptimes) Less(i, j int) bool {
return ts[i] < ts[j]
}
func (ts uptimes) Swap(i, j int) {
ts[i], ts[j] = ts[j], ts[i]
}
// getPeerUptimes - returns the uptime.
func getPeerUptimes(serverInfo []ServerInfo) time.Duration {
// In a single node Erasure or FS backend setup the uptime of
// the setup is the uptime of the single minio server
// instance.
if !globalIsDistXL {
return UTCNow().Sub(globalBootTime)
}
var times []time.Duration
for _, info := range serverInfo {
if info.Error != "" {
continue
}
times = append(times, info.Data.Properties.Uptime)
}
// Sort uptimes in chronological order.
sort.Sort(uptimes(times))
// Return the latest time as the uptime.
return times[0]
}
// GetLocksHandler - returns list of older lock from the server.
func (s *peerRESTServer) GetLocksHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "GetLocks")
locks := globalLockServer.ll.DupLockMap()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(locks))
w.(http.Flusher).Flush()
}
// LoadUsersHandler - returns server info.
func (s *peerRESTServer) LoadUsersHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
objAPI := newObjectLayerFn()
if objAPI == nil {
s.writeErrorResponse(w, errServerNotInitialized)
return
}
err := globalIAMSys.Load(objAPI)
if err != nil {
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
// StartProfilingHandler - Issues the start profiling command.
func (s *peerRESTServer) StartProfilingHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
profiler := vars[peerRESTProfiler]
if profiler == "" {
s.writeErrorResponse(w, errors.New("profiler name is missing"))
return
}
if globalProfiler != nil {
globalProfiler.Stop()
}
var err error
globalProfiler, err = startProfiler(profiler, "")
if err != nil {
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
// ServerInfoHandler - returns server info.
func (s *peerRESTServer) ServerInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "ServerInfo")
info, err := getServerInfo()
if err != nil {
s.writeErrorResponse(w, err)
return
}
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// DownloadProflingDataHandler - returns proflied data.
func (s *peerRESTServer) DownloadProflingDataHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "DownloadProfiling")
profileData, err := getProfileData()
if err != nil {
s.writeErrorResponse(w, err)
return
}
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(profileData))
}
// CPULoadInfoHandler - returns CPU Load info.
func (s *peerRESTServer) CPULoadInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "CPULoadInfo")
info := localEndpointsCPULoad(globalEndpoints)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// DrivePerfInfoHandler - returns Drive Performance info.
func (s *peerRESTServer) DrivePerfInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "DrivePerfInfo")
info := localEndpointsDrivePerf(globalEndpoints)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// MemUsageInfoHandler - returns Memory Usage info.
func (s *peerRESTServer) MemUsageInfoHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "MemUsageInfo")
info := localEndpointsMemUsage(globalEndpoints)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(info))
}
// DeleteBucketHandler - Delete notification and policies related to the bucket.
func (s *peerRESTServer) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
globalNotificationSys.RemoveNotification(bucketName)
globalPolicySys.Remove(bucketName)
w.(http.Flusher).Flush()
}
// ReloadFormatHandler - Reload Format.
func (s *peerRESTServer) ReloadFormatHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
dryRunString := vars[peerRESTDryRun]
if dryRunString == "" {
s.writeErrorResponse(w, errors.New("dry run parameter is missing"))
return
}
var dryRun bool
switch strings.ToLower(dryRunString) {
case "true":
dryRun = true
case "false":
dryRun = false
default:
s.writeErrorResponse(w, errInvalidArgument)
return
}
objAPI := newObjectLayerFn()
if objAPI == nil {
s.writeErrorResponse(w, errServerNotInitialized)
return
}
err := objAPI.ReloadFormat(context.Background(), dryRun)
if err != nil {
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
// RemoveBucketPolicyHandler - Remove bucket policy.
func (s *peerRESTServer) RemoveBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
globalPolicySys.Remove(bucketName)
w.(http.Flusher).Flush()
}
// SetBucketPolicyHandler - Set bucket policy.
func (s *peerRESTServer) SetBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var policyData policy.Policy
if r.ContentLength < 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&policyData)
if err != nil {
s.writeErrorResponse(w, err)
return
}
globalPolicySys.Set(bucketName, policyData)
w.(http.Flusher).Flush()
}
type remoteTargetExistsResp struct {
Exists bool
}
// TargetExistsHandler - Check if Target exists.
func (s *peerRESTServer) TargetExistsHandler(w http.ResponseWriter, r *http.Request) {
ctx := newContext(r, w, "TargetExists")
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var targetID event.TargetID
if r.ContentLength <= 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&targetID)
if err != nil {
s.writeErrorResponse(w, err)
return
}
var targetExists remoteTargetExistsResp
targetExists.Exists = globalNotificationSys.RemoteTargetExist(bucketName, targetID)
defer w.(http.Flusher).Flush()
logger.LogIf(ctx, gob.NewEncoder(w).Encode(&targetExists))
}
type sendEventRequest struct {
Event event.Event
TargetID event.TargetID
}
type sendEventResp struct {
Success bool
}
// SendEventHandler - Send Event.
func (s *peerRESTServer) SendEventHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
ctx := newContext(r, w, "SendEvent")
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var eventReq sendEventRequest
if r.ContentLength <= 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&eventReq)
if err != nil {
s.writeErrorResponse(w, err)
return
}
var eventResp sendEventResp
eventResp.Success = true
errs := globalNotificationSys.send(bucketName, eventReq.Event, eventReq.TargetID)
for i := range errs {
reqInfo := (&logger.ReqInfo{}).AppendTags("Event", eventReq.Event.EventName.String())
reqInfo.AppendTags("targetName", eventReq.TargetID.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, errs[i].Err)
eventResp.Success = false
s.writeErrorResponse(w, errs[i].Err)
return
}
logger.LogIf(ctx, gob.NewEncoder(w).Encode(&eventResp))
w.(http.Flusher).Flush()
}
// PutBucketNotificationHandler - Set bucket policy.
func (s *peerRESTServer) PutBucketNotificationHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var rulesMap event.RulesMap
if r.ContentLength < 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&rulesMap)
if err != nil {
s.writeErrorResponse(w, err)
return
}
globalNotificationSys.AddRulesMap(bucketName, rulesMap)
w.(http.Flusher).Flush()
}
type listenBucketNotificationReq struct {
EventNames []event.Name `json:"eventNames"`
Pattern string `json:"pattern"`
TargetID event.TargetID `json:"targetId"`
Addr xnet.Host `json:"addr"`
}
// ListenBucketNotificationHandler - Listen bucket notification handler.
func (s *peerRESTServer) ListenBucketNotificationHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
bucketName := vars[peerRESTBucket]
if bucketName == "" {
s.writeErrorResponse(w, errors.New("Bucket name is missing"))
return
}
var args listenBucketNotificationReq
if r.ContentLength <= 0 {
s.writeErrorResponse(w, errInvalidArgument)
return
}
err := gob.NewDecoder(r.Body).Decode(&args)
if err != nil {
s.writeErrorResponse(w, err)
return
}
restClient, err := newPeerRESTClient(&args.Addr)
if err != nil {
s.writeErrorResponse(w, fmt.Errorf("unable to find PeerRESTClient for provided address %v. This happens only if remote and this minio run with different set of endpoints", args.Addr))
return
}
target := NewPeerRESTClientTarget(bucketName, args.TargetID, restClient)
rulesMap := event.NewRulesMap(args.EventNames, args.Pattern, target.ID())
if err := globalNotificationSys.AddRemoteTarget(bucketName, target, rulesMap); err != nil {
reqInfo := &logger.ReqInfo{BucketName: target.bucketName}
reqInfo.AppendTags("target", target.id.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
s.writeErrorResponse(w, err)
return
}
w.(http.Flusher).Flush()
}
var errUnsupportedSignal = fmt.Errorf("unsupported signal: only restart and stop signals are supported")
// SignalServiceHandler - signal service handler.
func (s *peerRESTServer) SignalServiceHandler(w http.ResponseWriter, r *http.Request) {
if !s.IsValid(w, r) {
s.writeErrorResponse(w, errors.New("Invalid request"))
return
}
vars := mux.Vars(r)
signalString := vars[peerRESTSignal]
if signalString == "" {
s.writeErrorResponse(w, errors.New("signal name is missing"))
return
}
signal := serviceSignal(signalString)
defer w.(http.Flusher).Flush()
switch signal {
case serviceRestart, serviceStop:
globalServiceSignalCh <- signal
default:
s.writeErrorResponse(w, errUnsupportedSignal)
return
}
}
func (s *peerRESTServer) writeErrorResponse(w http.ResponseWriter, err error) {
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(err.Error()))
}
// IsValid - To authenticate and verify the time difference.
func (s *peerRESTServer) IsValid(w http.ResponseWriter, r *http.Request) bool {
if err := storageServerRequestValidate(r); err != nil {
s.writeErrorResponse(w, err)
return false
}
return true
}
// registerPeerRESTHandlers - register peer rest router.
func registerPeerRESTHandlers(router *mux.Router) {
server := &peerRESTServer{}
subrouter := router.PathPrefix(peerRESTPath).Subrouter()
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodGetLocks).HandlerFunc(httpTraceHdrs(server.GetLocksHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodServerInfo).HandlerFunc(httpTraceHdrs(server.ServerInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodCPULoadInfo).HandlerFunc(httpTraceHdrs(server.CPULoadInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodMemUsageInfo).HandlerFunc(httpTraceHdrs(server.MemUsageInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodDrivePerfInfo).HandlerFunc(httpTraceHdrs(server.DrivePerfInfoHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodDeleteBucket).HandlerFunc(httpTraceHdrs(server.DeleteBucketHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodSignalService).HandlerFunc(httpTraceHdrs(server.SignalServiceHandler)).Queries(restQueries(peerRESTSignal)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketPolicyRemove).HandlerFunc(httpTraceAll(server.RemoveBucketPolicyHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketPolicySet).HandlerFunc(httpTraceHdrs(server.SetBucketPolicyHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodLoadUsers).HandlerFunc(httpTraceAll(server.LoadUsersHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodStartProfiling).HandlerFunc(httpTraceAll(server.StartProfilingHandler)).Queries(restQueries(peerRESTProfiler)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodDownloadProfilingData).HandlerFunc(httpTraceHdrs(server.DownloadProflingDataHandler))
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodTargetExists).HandlerFunc(httpTraceHdrs(server.TargetExistsHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodSendEvent).HandlerFunc(httpTraceHdrs(server.SendEventHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketNotificationPut).HandlerFunc(httpTraceHdrs(server.PutBucketNotificationHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodBucketNotificationListen).HandlerFunc(httpTraceHdrs(server.ListenBucketNotificationHandler)).Queries(restQueries(peerRESTBucket)...)
subrouter.Methods(http.MethodPost).Path("/" + peerRESTMethodReloadFormat).HandlerFunc(httpTraceHdrs(server.ReloadFormatHandler)).Queries(restQueries(peerRESTDryRun)...)
router.NotFoundHandler = http.HandlerFunc(httpTraceAll(notFoundHandler))
}
-261
View File
@@ -1,261 +0,0 @@
/*
* Minio Cloud Storage, (C) 2018, 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"crypto/tls"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
// PeerRPCClient - peer RPC client talks to peer RPC server.
type PeerRPCClient struct {
*RPCClient
}
// DeleteBucket - calls delete bucket RPC.
func (rpcClient *PeerRPCClient) DeleteBucket(bucketName string) error {
args := DeleteBucketArgs{BucketName: bucketName}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".DeleteBucket", &args, &reply)
}
// SetBucketPolicy - calls set bucket policy RPC.
func (rpcClient *PeerRPCClient) SetBucketPolicy(bucketName string, bucketPolicy *policy.Policy) error {
args := SetBucketPolicyArgs{
BucketName: bucketName,
Policy: *bucketPolicy,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".SetBucketPolicy", &args, &reply)
}
// RemoveBucketPolicy - calls remove bucket policy RPC.
func (rpcClient *PeerRPCClient) RemoveBucketPolicy(bucketName string) error {
args := RemoveBucketPolicyArgs{
BucketName: bucketName,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".RemoveBucketPolicy", &args, &reply)
}
// PutBucketNotification - calls put bukcet notification RPC.
func (rpcClient *PeerRPCClient) PutBucketNotification(bucketName string, rulesMap event.RulesMap) error {
args := PutBucketNotificationArgs{
BucketName: bucketName,
RulesMap: rulesMap,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".PutBucketNotification", &args, &reply)
}
// ListenBucketNotification - calls listen bucket notification RPC.
func (rpcClient *PeerRPCClient) ListenBucketNotification(bucketName string, eventNames []event.Name,
pattern string, targetID event.TargetID, addr xnet.Host) error {
args := ListenBucketNotificationArgs{
BucketName: bucketName,
EventNames: eventNames,
Pattern: pattern,
TargetID: targetID,
Addr: addr,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".ListenBucketNotification", &args, &reply)
}
// RemoteTargetExist - calls remote target ID exist RPC.
func (rpcClient *PeerRPCClient) RemoteTargetExist(bucketName string, targetID event.TargetID) (bool, error) {
args := RemoteTargetExistArgs{
BucketName: bucketName,
TargetID: targetID,
}
var reply bool
err := rpcClient.Call(peerServiceName+".RemoteTargetExist", &args, &reply)
return reply, err
}
// SendEvent - calls send event RPC.
func (rpcClient *PeerRPCClient) SendEvent(bucketName string, targetID, remoteTargetID event.TargetID, eventData event.Event) error {
args := SendEventArgs{
BucketName: bucketName,
TargetID: remoteTargetID,
Event: eventData,
}
var reply bool
err := rpcClient.Call(peerServiceName+".SendEvent", &args, &reply)
if err != nil && !reply {
reqInfo := &logger.ReqInfo{BucketName: bucketName}
reqInfo.AppendTags("targetID", targetID.Name)
reqInfo.AppendTags("event", eventData.EventName.String())
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
globalNotificationSys.RemoveRemoteTarget(bucketName, targetID)
}
return err
}
// ReloadFormat - calls reload format RPC.
func (rpcClient *PeerRPCClient) ReloadFormat(dryRun bool) error {
args := ReloadFormatArgs{
DryRun: dryRun,
}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".ReloadFormat", &args, &reply)
}
// LoadUsers - calls load users RPC.
func (rpcClient *PeerRPCClient) LoadUsers() error {
args := AuthArgs{}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".LoadUsers", &args, &reply)
}
// LoadCredentials - calls load credentials RPC.
func (rpcClient *PeerRPCClient) LoadCredentials() error {
args := AuthArgs{}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".LoadCredentials", &args, &reply)
}
// DrivePerfInfo - returns drive performance info for remote server.
func (rpcClient *PeerRPCClient) DrivePerfInfo() (ServerDrivesPerfInfo, error) {
args := AuthArgs{}
var reply ServerDrivesPerfInfo
err := rpcClient.Call(peerServiceName+".DrivePerfInfo", &args, &reply)
return reply, err
}
// MemUsageInfo - returns mem utilization info for remote server
func (rpcClient *PeerRPCClient) MemUsageInfo() (ServerMemUsageInfo, error) {
args := AuthArgs{}
var reply ServerMemUsageInfo
err := rpcClient.Call(peerServiceName+".MemUsageInfo", &args, &reply)
return reply, err
}
// CPULoadInfo - returns cpu performance info for remote server
func (rpcClient *PeerRPCClient) CPULoadInfo() (ServerCPULoadInfo, error) {
args := AuthArgs{}
var reply ServerCPULoadInfo
err := rpcClient.Call(peerServiceName+".CPULoadInfo", &args, &reply)
return reply, err
}
// StartProfiling - starts profiling on the remote server.
func (rpcClient *PeerRPCClient) StartProfiling(profiler string) error {
args := StartProfilingArgs{Profiler: profiler}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".StartProfiling", &args, &reply)
}
// DownloadProfilingData - download already running profiling on the remote server.
func (rpcClient *PeerRPCClient) DownloadProfilingData() ([]byte, error) {
args := AuthArgs{}
var reply []byte
err := rpcClient.Call(peerServiceName+".DownloadProfilingData", &args, &reply)
return reply, err
}
// SignalService - calls load server info RPC.
func (rpcClient *PeerRPCClient) SignalService(sig serviceSignal) error {
args := SignalServiceArgs{Sig: sig}
reply := VoidReply{}
return rpcClient.Call(peerServiceName+".SignalService", &args, &reply)
}
// ServerInfo - calls load server info RPC.
func (rpcClient *PeerRPCClient) ServerInfo() (ServerInfoData, error) {
args := AuthArgs{}
reply := ServerInfoData{}
err := rpcClient.Call(peerServiceName+".ServerInfo", &args, &reply)
return reply, err
}
// GetLocksResp stores various info from the client for each lock that is requested.
type GetLocksResp map[string][]lockRequesterInfo
// GetLocks - returns the lock information on the server to which the RPC call is made.
func (rpcClient *PeerRPCClient) GetLocks() (resp GetLocksResp, err error) {
err = rpcClient.Call(peerServiceName+".GetLocks", &AuthArgs{}, &resp)
return resp, err
}
// NewPeerRPCClient - returns new peer RPC client.
func NewPeerRPCClient(host *xnet.Host) (*PeerRPCClient, error) {
scheme := "http"
if globalIsSSL {
scheme = "https"
}
serviceURL := &xnet.URL{
Scheme: scheme,
Host: host.String(),
Path: peerServicePath,
}
var tlsConfig *tls.Config
if globalIsSSL {
tlsConfig = &tls.Config{
ServerName: host.Name,
RootCAs: globalRootCAs,
}
}
rpcClient, err := NewRPCClient(
RPCClientArgs{
NewAuthTokenFunc: newAuthToken,
RPCVersion: globalRPCAPIVersion,
ServiceName: peerServiceName,
ServiceURL: serviceURL,
TLSConfig: tlsConfig,
},
)
if err != nil {
return nil, err
}
return &PeerRPCClient{rpcClient}, nil
}
// makeRemoteRPCClients - creates Peer RPCClients for given endpoint list.
func makeRemoteRPCClients(endpoints EndpointList) map[xnet.Host]*PeerRPCClient {
peerRPCClientMap := make(map[xnet.Host]*PeerRPCClient)
for _, hostStr := range GetRemotePeers(endpoints) {
host, err := xnet.ParseHost(hostStr)
logger.FatalIf(err, "Unable to parse peer RPC Host")
rpcClient, err := NewPeerRPCClient(host)
logger.FatalIf(err, "Unable to parse peer RPC Client")
peerRPCClientMap[*host] = rpcClient
}
return peerRPCClientMap
}
-419
View File
@@ -1,419 +0,0 @@
/*
* Minio Cloud Storage, (C) 2018, 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
import (
"context"
"fmt"
"path"
"sort"
"time"
"github.com/gorilla/mux"
"github.com/minio/minio/cmd/logger"
xrpc "github.com/minio/minio/cmd/rpc"
"github.com/minio/minio/pkg/event"
xnet "github.com/minio/minio/pkg/net"
"github.com/minio/minio/pkg/policy"
)
const peerServiceName = "Peer"
const peerServiceSubPath = "/peer/remote"
var peerServicePath = path.Join(minioReservedBucketPath, peerServiceSubPath)
// peerRPCReceiver - Peer RPC receiver for peer RPC server.
type peerRPCReceiver struct{}
// DeleteBucketArgs - delete bucket RPC arguments.
type DeleteBucketArgs struct {
AuthArgs
BucketName string
}
// DeleteBucket - handles delete bucket RPC call which removes all values of given bucket in global NotificationSys object.
func (receiver *peerRPCReceiver) DeleteBucket(args *DeleteBucketArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalNotificationSys.RemoveNotification(args.BucketName)
globalPolicySys.Remove(args.BucketName)
return nil
}
// SetBucketPolicyArgs - set bucket policy RPC arguments.
type SetBucketPolicyArgs struct {
AuthArgs
BucketName string
Policy policy.Policy
}
// SetBucketPolicy - handles set bucket policy RPC call which adds bucket policy to globalPolicySys.
func (receiver *peerRPCReceiver) SetBucketPolicy(args *SetBucketPolicyArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalPolicySys.Set(args.BucketName, args.Policy)
return nil
}
// RemoveBucketPolicyArgs - delete bucket policy RPC arguments.
type RemoveBucketPolicyArgs struct {
AuthArgs
BucketName string
}
// RemoveBucketPolicy - handles delete bucket policy RPC call which removes bucket policy to globalPolicySys.
func (receiver *peerRPCReceiver) RemoveBucketPolicy(args *RemoveBucketPolicyArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalPolicySys.Remove(args.BucketName)
return nil
}
// PutBucketNotificationArgs - put bucket notification RPC arguments.
type PutBucketNotificationArgs struct {
AuthArgs
BucketName string
RulesMap event.RulesMap
}
// PutBucketNotification - handles put bucket notification RPC call which adds rules to given bucket to global NotificationSys object.
func (receiver *peerRPCReceiver) PutBucketNotification(args *PutBucketNotificationArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
globalNotificationSys.AddRulesMap(args.BucketName, args.RulesMap)
return nil
}
// ListenBucketNotificationArgs - listen bucket notification RPC arguments.
type ListenBucketNotificationArgs struct {
AuthArgs `json:"-"`
BucketName string `json:"-"`
EventNames []event.Name `json:"eventNames"`
Pattern string `json:"pattern"`
TargetID event.TargetID `json:"targetId"`
Addr xnet.Host `json:"addr"`
}
// ListenBucketNotification - handles listen bucket notification RPC call.
// It creates PeerRPCClient target which pushes requested events to target in remote peer.
func (receiver *peerRPCReceiver) ListenBucketNotification(args *ListenBucketNotificationArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
rpcClient := globalNotificationSys.GetPeerRPCClient(args.Addr)
if rpcClient == nil {
return fmt.Errorf("unable to find PeerRPCClient for provided address %v. This happens only if remote and this minio run with different set of endpoints", args.Addr)
}
target := NewPeerRPCClientTarget(args.BucketName, args.TargetID, rpcClient)
rulesMap := event.NewRulesMap(args.EventNames, args.Pattern, target.ID())
if err := globalNotificationSys.AddRemoteTarget(args.BucketName, target, rulesMap); err != nil {
reqInfo := &logger.ReqInfo{BucketName: target.bucketName}
reqInfo.AppendTags("target", target.id.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, err)
return err
}
return nil
}
// RemoteTargetExistArgs - remote target ID exist RPC arguments.
type RemoteTargetExistArgs struct {
AuthArgs
BucketName string
TargetID event.TargetID
}
// RemoteTargetExist - handles target ID exist RPC call which checks whether given target ID is a HTTP client target or not.
func (receiver *peerRPCReceiver) RemoteTargetExist(args *RemoteTargetExistArgs, reply *bool) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = globalNotificationSys.RemoteTargetExist(args.BucketName, args.TargetID)
return nil
}
// SendEventArgs - send event RPC arguments.
type SendEventArgs struct {
AuthArgs
Event event.Event
TargetID event.TargetID
BucketName string
}
// SendEvent - handles send event RPC call which sends given event to target by given target ID.
func (receiver *peerRPCReceiver) SendEvent(args *SendEventArgs, reply *bool) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
// Set default to true to keep the target.
*reply = true
errs := globalNotificationSys.send(args.BucketName, args.Event, args.TargetID)
for i := range errs {
reqInfo := (&logger.ReqInfo{}).AppendTags("Event", args.Event.EventName.String())
reqInfo.AppendTags("targetName", args.TargetID.Name)
ctx := logger.SetReqInfo(context.Background(), reqInfo)
logger.LogIf(ctx, errs[i].Err)
*reply = false // send failed i.e. do not keep the target.
return errs[i].Err
}
return nil
}
// ReloadFormatArgs - send event RPC arguments.
type ReloadFormatArgs struct {
AuthArgs
DryRun bool
}
// ReloadFormat - handles reload format RPC call, reloads latest `format.json`
func (receiver *peerRPCReceiver) ReloadFormat(args *ReloadFormatArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
return objAPI.ReloadFormat(context.Background(), args.DryRun)
}
// LoadUsers - handles load users RPC call.
func (receiver *peerRPCReceiver) LoadUsers(args *AuthArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
return globalIAMSys.Load(objAPI)
}
// LoadCredentials - handles load credentials RPC call.
func (receiver *peerRPCReceiver) LoadCredentials(args *AuthArgs, reply *VoidReply) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
// Construct path to config.json for the given bucket.
configFile := path.Join(bucketConfigPrefix, minioConfigFile)
transactionConfigFile := configFile + ".transaction"
// As object layer's GetObject() and PutObject() take respective lock on minioMetaBucket
// and configFile, take a transaction lock to avoid race.
objLock := globalNSMutex.NewNSLock(minioMetaBucket, transactionConfigFile)
if err := objLock.GetRLock(globalOperationTimeout); err != nil {
return err
}
objLock.RUnlock()
return globalConfigSys.Load(newObjectLayerFn())
}
// DrivePerfInfo - handles drive performance RPC call
func (receiver *peerRPCReceiver) DrivePerfInfo(args *AuthArgs, reply *ServerDrivesPerfInfo) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = localEndpointsDrivePerf(globalEndpoints)
return nil
}
// CPULoadInfo - handles cpu performance RPC call
func (receiver *peerRPCReceiver) CPULoadInfo(args *AuthArgs, reply *ServerCPULoadInfo) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = localEndpointsCPULoad(globalEndpoints)
return nil
}
// MemUsageInfo - handles mem utilization RPC call
func (receiver *peerRPCReceiver) MemUsageInfo(args *AuthArgs, reply *ServerMemUsageInfo) error {
objAPI := newObjectLayerFn()
if objAPI == nil {
return errServerNotInitialized
}
*reply = localEndpointsMemUsage(globalEndpoints)
return nil
}
// uptimes - used to sort uptimes in chronological order.
type uptimes []time.Duration
func (ts uptimes) Len() int {
return len(ts)
}
func (ts uptimes) Less(i, j int) bool {
return ts[i] < ts[j]
}
func (ts uptimes) Swap(i, j int) {
ts[i], ts[j] = ts[j], ts[i]
}
// getPeerUptimes - returns the uptime.
func getPeerUptimes(serverInfo []ServerInfo) time.Duration {
// In a single node Erasure or FS backend setup the uptime of
// the setup is the uptime of the single minio server
// instance.
if !globalIsDistXL {
return UTCNow().Sub(globalBootTime)
}
var times []time.Duration
for _, info := range serverInfo {
if info.Error != "" {
continue
}
times = append(times, info.Data.Properties.Uptime)
}
// Sort uptimes in chronological order.
sort.Sort(uptimes(times))
// Return the latest time as the uptime.
return times[0]
}
// StartProfilingArgs - holds the RPC argument for StartingProfiling RPC call
type StartProfilingArgs struct {
AuthArgs
Profiler string
}
// StartProfiling - profiling server receiver.
func (receiver *peerRPCReceiver) StartProfiling(args *StartProfilingArgs, reply *VoidReply) error {
if globalProfiler != nil {
globalProfiler.Stop()
}
var err error
globalProfiler, err = startProfiler(args.Profiler, "")
return err
}
// DownloadProfilingData - download profiling data.
func (receiver *peerRPCReceiver) DownloadProfilingData(args *AuthArgs, reply *[]byte) error {
var err error
*reply, err = getProfileData()
return err
}
var errUnsupportedSignal = fmt.Errorf("unsupported signal: only restart and stop signals are supported")
// SignalServiceArgs - send event RPC arguments.
type SignalServiceArgs struct {
AuthArgs
Sig serviceSignal
}
// SignalService - signal service receiver.
func (receiver *peerRPCReceiver) SignalService(args *SignalServiceArgs, reply *VoidReply) error {
switch args.Sig {
case serviceRestart, serviceStop:
globalServiceSignalCh <- args.Sig
default:
return errUnsupportedSignal
}
return nil
}
// ServerInfo - server info receiver.
func (receiver *peerRPCReceiver) ServerInfo(args *AuthArgs, reply *ServerInfoData) error {
if globalBootTime.IsZero() {
return errServerNotInitialized
}
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
return errServerNotInitialized
}
// Server info data.
*reply = ServerInfoData{
StorageInfo: objLayer.StorageInfo(context.Background()),
ConnStats: globalConnStats.toServerConnStats(),
HTTPStats: globalHTTPStats.toServerHTTPStats(),
Properties: ServerProperties{
Uptime: UTCNow().Sub(globalBootTime),
Version: Version,
CommitID: CommitID,
SQSARN: globalNotificationSys.GetARNList(),
Region: globalServerConfig.GetRegion(),
},
}
return nil
}
// GetLocks - Get Locks receiver.
func (receiver *peerRPCReceiver) GetLocks(args *AuthArgs, reply *GetLocksResp) error {
if globalBootTime.IsZero() {
return errServerNotInitialized
}
// Build storage info
objLayer := newObjectLayerFn()
if objLayer == nil {
return errServerNotInitialized
}
// Locks data.
*reply = globalLockServer.ll.DupLockMap()
return nil
}
// NewPeerRPCServer - returns new peer RPC server.
func NewPeerRPCServer() (*xrpc.Server, error) {
rpcServer := xrpc.NewServer()
if err := rpcServer.RegisterName(peerServiceName, &peerRPCReceiver{}); err != nil {
return nil, err
}
return rpcServer, nil
}
// registerPeerRPCRouter - creates and registers Peer RPC server and its router.
func registerPeerRPCRouter(router *mux.Router) {
rpcServer, err := NewPeerRPCServer()
logger.FatalIf(err, "Unable to initialize peer RPC Server")
subrouter := router.PathPrefix(minioReservedBucketPath).Subrouter()
subrouter.Path(peerServiceSubPath).HandlerFunc(httpTraceHdrs(rpcServer.ServeHTTP))
}
+32 -34
View File
@@ -232,7 +232,6 @@ func getDiskInfo(diskPath string) (di disk.Info, err error) {
var ignoreDiskFreeOS = []string{
globalWindowsOSName,
globalNetBSDOSName,
globalSolarisOSName,
}
// check if disk total has minimum required size.
@@ -943,20 +942,6 @@ func (s *posix) openFile(volume, path string, mode int) (f *os.File, err error)
return w, nil
}
// Just like io.LimitedReader but supports Close() to be compatible with io.ReadCloser that is
// returned by posix.ReadFileStream()
type posixLimitedReader struct {
io.LimitedReader
}
func (l *posixLimitedReader) Close() error {
c, ok := l.R.(io.Closer)
if !ok {
return errUnexpected
}
return c.Close()
}
// ReadFileStream - Returns the read stream of the file.
func (s *posix) ReadFileStream(volume, path string, offset, length int64) (io.ReadCloser, error) {
var err error
@@ -1030,7 +1015,10 @@ func (s *posix) ReadFileStream(volume, path string, offset, length int64) (io.Re
if _, err = file.Seek(offset, io.SeekStart); err != nil {
return nil, err
}
return &posixLimitedReader{io.LimitedReader{R: file, N: length}}, nil
return struct {
io.Reader
io.Closer
}{Reader: io.LimitReader(file, length), Closer: file}, nil
}
// CreateFile - creates the file.
@@ -1195,18 +1183,19 @@ func (s *posix) StatFile(volume, path string) (file FileInfo, err error) {
}
st, err := os.Stat((filePath))
if err != nil {
// File is really not found.
if os.IsNotExist(err) {
switch {
case os.IsNotExist(err):
// File is really not found.
return FileInfo{}, errFileNotFound
} else if isSysErrIO(err) {
case isSysErrIO(err):
return FileInfo{}, errFaultyDisk
} else if isSysErrNotDir(err) {
case isSysErrNotDir(err):
// File path cannot be verified since one of the parents is a file.
return FileInfo{}, errFileNotFound
default:
// Return all errors here.
return FileInfo{}, err
}
// Return all errors here.
return FileInfo{}, err
}
// If its a directory its not a regular file.
if st.Mode().IsDir() {
@@ -1231,21 +1220,21 @@ func deleteFile(basePath, deletePath string) error {
// Attempt to remove path.
if err := os.Remove((deletePath)); err != nil {
// Ignore errors if the directory is not empty. The server relies on
// this functionality, and sometimes uses recursion that should not
// error on parent directories.
if isSysErrNotEmpty(err) {
switch {
case isSysErrNotEmpty(err):
// Ignore errors if the directory is not empty. The server relies on
// this functionality, and sometimes uses recursion that should not
// error on parent directories.
return nil
}
if os.IsNotExist(err) {
case os.IsNotExist(err):
return errFileNotFound
} else if os.IsPermission(err) {
case os.IsPermission(err):
return errFileAccessDenied
} else if isSysErrIO(err) {
case isSysErrIO(err):
return errFaultyDisk
default:
return err
}
return err
}
// Trailing slash is removed when found to ensure
@@ -1369,9 +1358,18 @@ func (s *posix) RenameFile(srcVolume, srcPath, dstVolume, dstPath string) (err e
if err == nil && !isDirEmpty(dstFilePath) {
return errFileAccessDenied
}
if !os.IsNotExist(err) {
if err != nil && !os.IsNotExist(err) {
return err
}
// Empty destination remove it before rename.
if isDirEmpty(dstFilePath) {
if err = os.Remove(dstFilePath); err != nil {
if isSysErrNotEmpty(err) {
return errFileAccessDenied
}
return err
}
}
}
if err = renameAll(srcFilePath, dstFilePath); err != nil {
+3 -2
View File
@@ -38,11 +38,12 @@ func registerDistXLRouters(router *mux.Router, endpoints EndpointList) {
// Register storage rpc router only if its a distributed setup.
registerStorageRESTHandlers(router, endpoints)
// Register peer REST router only if its a distributed setup.
registerPeerRESTHandlers(router)
// Register distributed namespace lock.
registerDistNSLockRouter(router)
// Register peer communication router.
registerPeerRPCRouter(router)
}
// List of some generic handlers which are applied for all incoming requests.
+8 -9
View File
@@ -19,15 +19,16 @@ package cmd
import (
"os"
"os/exec"
"syscall"
)
// Type of service signals currently supported.
type serviceSignal int
type serviceSignal string
const (
serviceStatus = iota // Gets status about the service.
serviceRestart // Restarts the service.
serviceStop // Stops the server.
serviceStatus serviceSignal = "serviceStatus" // Gets status about the service.
serviceRestart = "serviceRestart" // Restarts the service.
serviceStop = "serviceStop" // Stops the server.
// Add new service requests here.
)
@@ -56,9 +57,7 @@ func restartProcess() error {
return err
}
// Pass on the environment and replace the old count key with the new one.
cmd := exec.Command(argv0, os.Args[1:]...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd.Start()
// Invokes the execve system call.
// Re-uses the same pid. This preserves the pid over multiple server-respawns.
return syscall.Exec(argv0, os.Args, os.Environ())
}
+1 -1
View File
@@ -66,7 +66,7 @@ func getReqAccessKeyV4(r *http.Request, region string, stype serviceType) (auth.
// parse credentialHeader string into its structured form.
func parseCredentialHeader(credElement string, region string, stype serviceType) (ch credentialHeader, aec APIErrorCode) {
creds := strings.Split(strings.TrimSpace(credElement), "=")
creds := strings.SplitN(strings.TrimSpace(credElement), "=", 2)
if len(creds) != 2 {
return ch, ErrMissingFields
}
+19
View File
@@ -216,6 +216,25 @@ func TestParseCredentialHeader(t *testing.T) {
"aws4_request"),
expectedErrCode: ErrNone,
},
// Test Case - 11.
// Test case with right inputs -> AccessKey contains `=`. See minio/#7376
// "aws4_request" is the valid request version.
{
inputCredentialStr: generateCredentialStr(
"LOCALKEY/DEV/1=",
sampleTimeStr,
"us-west-1",
"s3",
"aws4_request"),
expectedCredentials: generateCredentials(
t,
"LOCALKEY/DEV/1=",
sampleTimeStr,
"us-west-1",
"s3",
"aws4_request"),
expectedErrCode: ErrNone,
},
}
for i, testCase := range testCases {
+4 -1
View File
@@ -66,9 +66,12 @@ var errVolumeNotEmpty = errors.New("volume is not empty")
// errVolumeAccessDenied - cannot access volume, insufficient permissions.
var errVolumeAccessDenied = errors.New("volume access denied")
// errVolumeAccessDenied - cannot access file, insufficient permissions.
// errFileAccessDenied - cannot access file, insufficient permissions.
var errFileAccessDenied = errors.New("file access denied")
// errFileParentIsFile - cannot have overlapping objects, parent is already a file.
var errFileParentIsFile = errors.New("parent is a file")
// errBitrotHashAlgoInvalid - the algo for bit-rot hash
// verification is empty or invalid.
var errBitrotHashAlgoInvalid = errors.New("bit-rot hash algorithm is invalid")
+1 -6
View File
@@ -25,7 +25,6 @@ import (
"net/url"
"path"
"strconv"
"time"
"encoding/gob"
"encoding/hex"
@@ -38,10 +37,6 @@ import (
xnet "github.com/minio/minio/pkg/net"
)
// The timeout of TCP connect and sending/receiving
// data for all internode storage REST requests.
const storageRESTTimeout = 5 * time.Minute
func isNetworkError(err error) bool {
if err == nil {
return false
@@ -405,7 +400,7 @@ func newStorageRESTClient(endpoint Endpoint) (*storageRESTClient, error) {
}
}
restClient, err := rest.NewClient(serverURL, tlsConfig, storageRESTTimeout, newAuthToken)
restClient, err := rest.NewClient(serverURL, tlsConfig, rest.DefaultRESTTimeout, newAuthToken)
if err != nil {
return nil, err
}
+10 -6
View File
@@ -17,17 +17,14 @@
package cmd
import (
"encoding/gob"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"path"
"strconv"
"net/http"
"encoding/gob"
"encoding/hex"
"time"
"github.com/gorilla/mux"
@@ -47,6 +44,7 @@ type storageRESTServer struct {
func (s *storageRESTServer) writeErrorResponse(w http.ResponseWriter, err error) {
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(err.Error()))
w.(http.Flusher).Flush()
}
// Authenticates storage client's requests and validates for skewed time.
@@ -93,6 +91,7 @@ func (s *storageRESTServer) GetInstanceID(w http.ResponseWriter, r *http.Request
}
w.Header().Set("Content-Length", strconv.Itoa(len(s.instanceID)))
w.Write([]byte(s.instanceID))
w.(http.Flusher).Flush()
}
// DiskInfoHandler - returns disk info.
@@ -268,6 +267,7 @@ func (s *storageRESTServer) ReadAllHandler(w http.ResponseWriter, r *http.Reques
}
w.Header().Set("Content-Length", strconv.Itoa(len(buf)))
w.Write(buf)
w.(http.Flusher).Flush()
}
// ReadFileHandler - read section of a file.
@@ -311,6 +311,7 @@ func (s *storageRESTServer) ReadFileHandler(w http.ResponseWriter, r *http.Reque
}
w.Header().Set("Content-Length", strconv.Itoa(len(buf)))
w.Write(buf)
w.(http.Flusher).Flush()
}
// ReadFileHandler - read section of a file.
@@ -331,6 +332,7 @@ func (s *storageRESTServer) ReadFileStreamHandler(w http.ResponseWriter, r *http
s.writeErrorResponse(w, err)
return
}
rc, err := s.storage.ReadFileStream(volume, filePath, int64(offset), int64(length))
if err != nil {
s.writeErrorResponse(w, err)
@@ -338,7 +340,9 @@ func (s *storageRESTServer) ReadFileStreamHandler(w http.ResponseWriter, r *http
}
defer rc.Close()
w.Header().Set("Content-Length", strconv.Itoa(length))
io.Copy(w, rc)
w.(http.Flusher).Flush()
}
// ListDirHandler - list a directory.
-1
View File
@@ -33,7 +33,6 @@ type listParams struct {
recursive bool
marker string
prefix string
heal bool
}
// errWalkAbort - returned by doTreeWalk() if it returns prematurely.
+28 -68
View File
@@ -22,7 +22,6 @@ import (
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"net/url"
"os"
@@ -39,6 +38,7 @@ import (
"github.com/gorilla/rpc/v2/json2"
miniogopolicy "github.com/minio/minio-go/pkg/policy"
"github.com/minio/minio-go/pkg/s3utils"
"github.com/minio/minio-go/pkg/set"
"github.com/minio/minio/browser"
"github.com/minio/minio/cmd/crypto"
"github.com/minio/minio/cmd/logger"
@@ -296,24 +296,29 @@ func (web *webAPIHandlers) ListBuckets(r *http.Request, args *WebGenericArgs, re
// If etcd, dns federation configured list buckets from etcd.
if globalDNSConfig != nil {
dnsBuckets, err := globalDNSConfig.List()
if err != nil {
if err != nil && err != dns.ErrNoEntriesFound {
return toJSONError(err)
}
bucketSet := set.NewStringSet()
for _, dnsRecord := range dnsBuckets {
bucketName := strings.Trim(dnsRecord.Key, "/")
if bucketSet.Contains(dnsRecord.Key) {
continue
}
if globalIAMSys.IsAllowed(iampolicy.Args{
AccountName: claims.Subject,
Action: iampolicy.ListBucketAction,
BucketName: bucketName,
BucketName: dnsRecord.Key,
ConditionValues: getConditionValues(r, "", claims.Subject),
IsOwner: owner,
ObjectName: "",
}) {
reply.Buckets = append(reply.Buckets, WebBucketInfo{
Name: bucketName,
Name: dnsRecord.Key,
CreationDate: dnsRecord.CreationDate,
})
bucketSet.Add(dnsRecord.Key)
}
}
} else {
@@ -758,20 +763,11 @@ func (web *webAPIHandlers) SetAuth(r *http.Request, args *SetAuthArgs, reply *Se
return toJSONError(err)
}
if errs := globalNotificationSys.LoadCredentials(); len(errs) != 0 {
reply.PeerErrMsgs = make(map[string]string)
for _, nerr := range errs {
err = fmt.Errorf("Unable to update credentials on server %v: %v", nerr.Host, nerr.Err)
logger.LogIf(context.Background(), err)
reply.PeerErrMsgs[nerr.Host.String()] = err.Error()
}
} else {
reply.Token, err = authenticateWeb(creds.AccessKey, creds.SecretKey)
if err != nil {
return toJSONError(err)
}
reply.UIVersion = browser.UIVersion
reply.Token, err = authenticateWeb(creds.AccessKey, creds.SecretKey)
if err != nil {
return toJSONError(err)
}
reply.UIVersion = browser.UIVersion
return nil
}
@@ -986,12 +982,6 @@ func (web *webAPIHandlers) Upload(w http.ResponseWriter, r *http.Request) {
}
}
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object created event.
sendEvent(eventArgs{
EventName: event.ObjectCreatedPut,
@@ -1000,8 +990,7 @@ func (web *webAPIHandlers) Upload(w http.ResponseWriter, r *http.Request) {
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -1119,12 +1108,6 @@ func (web *webAPIHandlers) Download(w http.ResponseWriter, r *http.Request) {
}
}
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
// Notify object accessed via a GET request.
sendEvent(eventArgs{
EventName: event.ObjectAccessedGet,
@@ -1133,8 +1116,7 @@ func (web *webAPIHandlers) Download(w http.ResponseWriter, r *http.Request) {
ReqParams: extractReqParams(r),
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
Host: handlers.GetSourceIP(r),
})
}
@@ -1149,11 +1131,7 @@ type DownloadZipArgs struct {
// Takes a list of objects and creates a zip file that sent as the response body.
func (web *webAPIHandlers) DownloadZip(w http.ResponseWriter, r *http.Request) {
// Get host and port from Request.RemoteAddr.
host, port, err := net.SplitHostPort(handlers.GetSourceIP(r))
if err != nil {
host, port = "", ""
}
host := handlers.GetSourceIP(r)
ctx := newContext(r, w, "WebDownloadZip")
defer logger.AuditLog(w, r, "WebDownloadZip", mustGetClaimsFromToken(r))
@@ -1338,7 +1316,6 @@ func (web *webAPIHandlers) DownloadZip(w http.ResponseWriter, r *http.Request) {
RespElements: extractRespElements(w),
UserAgent: r.UserAgent(),
Host: host,
Port: port,
})
return nil
@@ -1792,67 +1769,50 @@ func toJSONError(err error, params ...string) (jerr *json2.Error) {
// toWebAPIError - convert into error into APIError.
func toWebAPIError(err error) APIError {
if err == errAuthentication {
return APIError{
Code: "AccessDenied",
HTTPStatusCode: http.StatusForbidden,
Description: err.Error(),
}
} else if err == errServerNotInitialized {
switch err {
case errServerNotInitialized:
return APIError{
Code: "XMinioServerNotInitialized",
HTTPStatusCode: http.StatusServiceUnavailable,
Description: err.Error(),
}
} else if err == auth.ErrInvalidAccessKeyLength {
case errAuthentication, auth.ErrInvalidAccessKeyLength, auth.ErrInvalidSecretKeyLength, errInvalidAccessKeyID:
return APIError{
Code: "AccessDenied",
HTTPStatusCode: http.StatusForbidden,
Description: err.Error(),
}
} else if err == auth.ErrInvalidSecretKeyLength {
return APIError{
Code: "AccessDenied",
HTTPStatusCode: http.StatusForbidden,
Description: err.Error(),
}
} else if err == errInvalidAccessKeyID {
return APIError{
Code: "AccessDenied",
HTTPStatusCode: http.StatusForbidden,
Description: err.Error(),
}
} else if err == errSizeUnspecified {
case errSizeUnspecified:
return APIError{
Code: "InvalidRequest",
HTTPStatusCode: http.StatusBadRequest,
Description: err.Error(),
}
} else if err == errChangeCredNotAllowed {
case errChangeCredNotAllowed:
return APIError{
Code: "MethodNotAllowed",
HTTPStatusCode: http.StatusMethodNotAllowed,
Description: err.Error(),
}
} else if err == errInvalidBucketName {
case errInvalidBucketName:
return APIError{
Code: "InvalidBucketName",
HTTPStatusCode: http.StatusBadRequest,
Description: err.Error(),
}
} else if err == errInvalidArgument {
case errInvalidArgument:
return APIError{
Code: "InvalidArgument",
HTTPStatusCode: http.StatusBadRequest,
Description: err.Error(),
}
} else if err == errEncryptedObject {
case errEncryptedObject:
return getAPIError(ErrSSEEncryptedObject)
} else if err == errInvalidEncryptionParameters {
case errInvalidEncryptionParameters:
return getAPIError(ErrInvalidEncryptionParameters)
} else if err == errObjectTampered {
case errObjectTampered:
return getAPIError(ErrObjectTampered)
} else if err == errMethodNotAllowed {
case errMethodNotAllowed:
return getAPIError(ErrMethodNotAllowed)
}
+21 -96
View File
@@ -725,7 +725,7 @@ func (s *xlSets) ListObjects(ctx context.Context, bucket, prefix, marker, delimi
recursive = false
}
walkResultCh, endWalkCh := s.listPool.Release(listParams{bucket, recursive, marker, prefix, false})
walkResultCh, endWalkCh := s.listPool.Release(listParams{bucket, recursive, marker, prefix})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := func(bucket, entry string) bool {
@@ -798,7 +798,7 @@ func (s *xlSets) ListObjects(ctx context.Context, bucket, prefix, marker, delimi
}
}
params := listParams{bucket, recursive, nextMarker, prefix, false}
params := listParams{bucket, recursive, nextMarker, prefix}
if !eof {
s.listPool.Set(params, walkResultCh, endWalkCh)
}
@@ -1010,9 +1010,6 @@ func (s *xlSets) ReloadFormat(ctx context.Context, dryRun bool) (err error) {
// If it is a single node XL and all disks are root disks, it is most likely a test setup, else it is a production setup.
// On a test setup we allow creation of format.json on root disks to help with dev/testing.
func isTestSetup(infos []DiskInfo, errs []error) bool {
if globalIsDistXL {
return false
}
rootDiskCount := 0
for i := range errs {
if errs[i] != nil {
@@ -1299,8 +1296,8 @@ func (s *xlSets) HealBucket(ctx context.Context, bucket string, dryRun, remove b
}
// HealObject - heals inconsistent object on a hashedSet based on object name.
func (s *xlSets) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool) (madmin.HealResultItem, error) {
return s.getHashedSet(object).HealObject(ctx, bucket, object, dryRun, remove)
func (s *xlSets) HealObject(ctx context.Context, bucket, object string, dryRun, remove bool, scanMode madmin.HealScanMode) (madmin.HealResultItem, error) {
return s.getHashedSet(object).HealObject(ctx, bucket, object, dryRun, remove, scanMode)
}
// Lists all buckets which need healing.
@@ -1322,110 +1319,38 @@ func (s *xlSets) ListBucketsHeal(ctx context.Context) ([]BucketInfo, error) {
return listBuckets, nil
}
// listObjectsHeal - wrapper function implemented over file tree walk.
func (s *xlSets) listObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
// Default is recursive, if delimiter is set then list non recursive.
// HealObjects - Heal all objects recursively at a specified prefix, any
// dangling objects deleted as well automatically.
func (s *xlSets) HealObjects(ctx context.Context, bucket, prefix string, healObjectFn func(string, string) error) (err error) {
recursive := true
if delimiter == slashSeparator {
recursive = false
endWalkCh := make(chan struct{})
isLeaf := func(bucket, entry string) bool {
return hasSuffix(entry, xlMetaJSONFile)
}
// "heal" true for listObjectsHeal() and false for listObjects()
walkResultCh, endWalkCh := s.listPool.Release(listParams{bucket, recursive, marker, prefix, true})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := func(bucket, entry string) bool {
entry = strings.TrimSuffix(entry, slashSeparator)
// Verify if we are at the leaf, a leaf is where we
// see `xl.json` inside a directory.
return s.getHashedSet(entry).isObject(bucket, entry)
}
isLeafDir := func(bucket, entry string) bool {
var ok bool
for _, set := range s.sets {
ok = set.isObjectDir(bucket, entry)
if ok {
return true
}
}
return false
}
listDir := listDirSetsFactory(ctx, isLeaf, isLeafDir, s.sets...)
walkResultCh = startTreeWalk(ctx, bucket, prefix, marker, recursive, listDir, isLeaf, isLeafDir, endWalkCh)
isLeafDir := func(bucket, entry string) bool {
return false
}
var objInfos []ObjectInfo
var eof bool
var nextMarker string
for i := 0; i < maxKeys; {
listDir := listDirSetsFactory(ctx, isLeaf, isLeafDir, s.sets...)
walkResultCh := startTreeWalk(ctx, bucket, prefix, "", recursive, listDir, isLeaf, isLeafDir, endWalkCh)
for {
walkResult, ok := <-walkResultCh
if !ok {
// Closed channel.
eof = true
break
}
// For any walk error return right away.
if walkResult.err != nil {
return loi, toObjectErr(walkResult.err, bucket, prefix)
return toObjectErr(walkResult.err, bucket, prefix)
}
if err := healObjectFn(bucket, strings.TrimSuffix(walkResult.entry, slashSeparator+xlMetaJSONFile)); err != nil {
return toObjectErr(err, bucket, walkResult.entry)
}
var objInfo ObjectInfo
objInfo.Bucket = bucket
objInfo.Name = walkResult.entry
nextMarker = objInfo.Name
objInfos = append(objInfos, objInfo)
i++
if walkResult.end {
eof = true
break
}
}
params := listParams{bucket, recursive, nextMarker, prefix, true}
if !eof {
s.listPool.Set(params, walkResultCh, endWalkCh)
}
result := ListObjectsInfo{IsTruncated: !eof}
for _, objInfo := range objInfos {
result.NextMarker = objInfo.Name
result.Objects = append(result.Objects, objInfo)
}
return result, nil
}
// This is not implemented yet, will be implemented later to comply with Admin API refactor.
func (s *xlSets) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, err error) {
if err = checkListObjsArgs(ctx, bucket, prefix, marker, delimiter, s); err != nil {
return loi, err
}
// With max keys of zero we have reached eof, return right here.
if maxKeys == 0 {
return loi, nil
}
// For delimiter and prefix as '/' we do not list anything at all
// since according to s3 spec we stop at the 'delimiter' along
// with the prefix. On a flat namespace with 'prefix' as '/'
// we don't have any entries, since all the keys are of form 'keyName/...'
if delimiter == slashSeparator && prefix == slashSeparator {
return loi, nil
}
// Over flowing count - reset to maxObjectList.
if maxKeys < 0 || maxKeys > maxObjectList {
maxKeys = maxObjectList
}
// Initiate a list operation, if successful filter and return quickly.
listObjInfo, err := s.listObjectsHeal(ctx, bucket, prefix, marker, delimiter, maxKeys)
if err == nil {
// We got the entries successfully return.
return listObjInfo, nil
}
// Return error at the end.
return loi, toObjectErr(err, bucket, prefix)
return nil
}
+33 -21
View File
@@ -22,6 +22,7 @@ import (
"time"
"github.com/minio/minio/cmd/logger"
"github.com/minio/minio/pkg/madmin"
)
// commonTime returns a maximally occurring time from a list of time.
@@ -158,37 +159,48 @@ func getLatestXLMeta(ctx context.Context, partsMetadata []xlMetaV1, errs []error
// - slice of errors about the state of data files on disk - can have
// a not-found error or a hash-mismatch error.
func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetadata []xlMetaV1, errs []error, bucket,
object string) ([]StorageAPI, []error) {
object string, scanMode madmin.HealScanMode) ([]StorageAPI, []error) {
availableDisks := make([]StorageAPI, len(onlineDisks))
dataErrs := make([]error, len(onlineDisks))
for i, onlineDisk := range onlineDisks {
if onlineDisk == nil {
dataErrs[i] = errDiskNotFound
dataErrs[i] = errs[i]
continue
}
erasureInfo := partsMetadata[i].Erasure
erasure, err := NewErasure(ctx, erasureInfo.DataBlocks, erasureInfo.ParityBlocks, erasureInfo.BlockSize)
if err != nil {
dataErrs[i] = err
continue
}
// disk has a valid xl.json but may not have all the
// parts. This is considered an outdated disk, since
// it needs healing too.
for _, part := range partsMetadata[i].Parts {
checksumInfo := erasureInfo.GetChecksumInfo(part.Name)
tillOffset := erasure.ShardFileTillOffset(0, part.Size, part.Size)
err = bitrotCheckFile(onlineDisk, bucket, pathJoin(object, part.Name), tillOffset, checksumInfo.Algorithm, checksumInfo.Hash, erasure.ShardSize())
switch scanMode {
case madmin.HealDeepScan:
erasureInfo := partsMetadata[i].Erasure
erasure, err := NewErasure(ctx, erasureInfo.DataBlocks, erasureInfo.ParityBlocks, erasureInfo.BlockSize)
if err != nil {
isCorrupt := strings.HasPrefix(err.Error(), "Bitrot verification mismatch - expected ")
if !isCorrupt && err != errFileNotFound && err != errVolumeNotFound {
logger.LogIf(ctx, err)
}
dataErrs[i] = err
break
continue
}
// disk has a valid xl.json but may not have all the
// parts. This is considered an outdated disk, since
// it needs healing too.
for _, part := range partsMetadata[i].Parts {
checksumInfo := erasureInfo.GetChecksumInfo(part.Name)
tillOffset := erasure.ShardFileTillOffset(0, part.Size, part.Size)
err = bitrotCheckFile(onlineDisk, bucket, pathJoin(object, part.Name), tillOffset, checksumInfo.Algorithm, checksumInfo.Hash, erasure.ShardSize())
if err != nil {
isCorrupt := strings.HasPrefix(err.Error(), "Bitrot verification mismatch - expected ")
if !isCorrupt && err != errFileNotFound && err != errVolumeNotFound {
logger.LogIf(ctx, err)
}
dataErrs[i] = err
break
}
}
case madmin.HealNormalScan:
for _, part := range partsMetadata[i].Parts {
_, err := onlineDisk.StatFile(bucket, pathJoin(object, part.Name))
if err != nil {
dataErrs[i] = err
break
}
}
}
+5 -3
View File
@@ -23,6 +23,8 @@ import (
"path/filepath"
"testing"
"time"
"github.com/minio/minio/pkg/madmin"
)
// validates functionality provided to find most common
@@ -239,7 +241,7 @@ func TestListOnlineDisks(t *testing.T) {
i+1, test.expectedTime, modTime)
}
availableDisks, newErrs := disksWithAllParts(context.Background(), onlineDisks, partsMetadata, test.errs, bucket, object)
availableDisks, newErrs := disksWithAllParts(context.Background(), onlineDisks, partsMetadata, test.errs, bucket, object, madmin.HealDeepScan)
test.errs = newErrs
if test._tamperBackend != noTamper {
@@ -291,7 +293,7 @@ func TestDisksWithAllParts(t *testing.T) {
t.Fatalf("Failed to read xl meta data %v", err)
}
filteredDisks, errs := disksWithAllParts(ctx, xlDisks, partsMetadata, errs, bucket, object)
filteredDisks, errs := disksWithAllParts(ctx, xlDisks, partsMetadata, errs, bucket, object, madmin.HealDeepScan)
if len(filteredDisks) != len(xlDisks) {
t.Errorf("Unexpected number of disks: %d", len(filteredDisks))
@@ -328,7 +330,7 @@ func TestDisksWithAllParts(t *testing.T) {
}
errs = make([]error, len(xlDisks))
filteredDisks, errs = disksWithAllParts(ctx, xlDisks, partsMetadata, errs, bucket, object)
filteredDisks, errs = disksWithAllParts(ctx, xlDisks, partsMetadata, errs, bucket, object, madmin.HealDeepScan)
if len(filteredDisks) != len(xlDisks) {
t.Errorf("Unexpected number of disks: %d", len(filteredDisks))
+84 -47
View File
@@ -207,39 +207,29 @@ func shouldHealObjectOnDisk(xlErr, dataErr error, meta xlMetaV1, quorumModTime t
}
// Heals an object by re-writing corrupt/missing erasure blocks.
func healObject(ctx context.Context, storageDisks []StorageAPI, bucket string, object string,
quorum int, dryRun bool) (result madmin.HealResultItem, err error) {
partsMetadata, errs := readAllXLMetadata(ctx, storageDisks, bucket, object)
func (xl xlObjects) healObject(ctx context.Context, bucket string, object string,
partsMetadata []xlMetaV1, errs []error, latestXLMeta xlMetaV1,
dryRun bool, remove bool, scanMode madmin.HealScanMode) (result madmin.HealResultItem, err error) {
errCount := 0
for _, err := range errs {
if err != nil {
errCount++
}
}
dataBlocks := latestXLMeta.Erasure.DataBlocks
if errCount == len(errs) {
// Only if we get errors from all the disks we return error. Else we need to
// continue to return filled madmin.HealResultItem struct which includes info
// on what disks the file is available etc.
if reducedErr := reduceReadQuorumErrs(ctx, errs, nil, quorum); reducedErr != nil {
return defaultHealResult(storageDisks, errs, bucket, object), toObjectErr(reducedErr, bucket, object)
}
}
storageDisks := xl.getDisks()
// List of disks having latest version of the object xl.json
// (by modtime).
latestDisks, modTime := listOnlineDisks(storageDisks, partsMetadata, errs)
// List of disks having all parts as per latest xl.json.
availableDisks, dataErrs := disksWithAllParts(ctx, latestDisks, partsMetadata, errs, bucket, object)
availableDisks, dataErrs := disksWithAllParts(ctx, latestDisks, partsMetadata, errs, bucket, object, scanMode)
// Initialize heal result object
result = madmin.HealResultItem{
Type: madmin.HealItemObject,
Bucket: bucket,
Object: object,
DiskCount: len(storageDisks),
Type: madmin.HealItemObject,
Bucket: bucket,
Object: object,
DiskCount: len(storageDisks),
ParityBlocks: latestXLMeta.Erasure.ParityBlocks,
DataBlocks: latestXLMeta.Erasure.DataBlocks,
// Initialize object size to -1, so we can detect if we are
// unable to reliably find the object size.
@@ -307,11 +297,18 @@ func healObject(ctx context.Context, storageDisks []StorageAPI, bucket string, o
// If less than read quorum number of disks have all the parts
// of the data, we can't reconstruct the erasure-coded data.
if numAvailableDisks < quorum {
// Default to most common configuration for erasure
// blocks upon returning quorum error.
result.ParityBlocks = len(storageDisks) / 2
result.DataBlocks = len(storageDisks) / 2
if numAvailableDisks < dataBlocks {
// Check if xl.json, and corresponding parts are also missing.
if m, ok := isObjectDangling(partsMetadata, errs, dataErrs); ok {
writeQuorum := m.Erasure.DataBlocks + 1
if m.Erasure.DataBlocks == 0 {
writeQuorum = len(storageDisks)/2 + 1
}
if !dryRun && remove {
err = xl.deleteObject(ctx, bucket, object, writeQuorum, false)
}
return defaultHealResult(latestXLMeta, storageDisks, errs, bucket, object), err
}
return result, toObjectErr(errXLReadQuorum, bucket, object)
}
@@ -328,7 +325,7 @@ func healObject(ctx context.Context, storageDisks []StorageAPI, bucket string, o
// Latest xlMetaV1 for reference. If a valid metadata is not
// present, it is as good as object not found.
latestMeta, pErr := pickValidXLMeta(ctx, partsMetadata, modTime, quorum)
latestMeta, pErr := pickValidXLMeta(ctx, partsMetadata, modTime, dataBlocks)
if pErr != nil {
return result, toObjectErr(pErr, bucket, object)
}
@@ -533,7 +530,7 @@ func (xl xlObjects) healObjectDir(ctx context.Context, bucket, object string, dr
// Populates default heal result item entries with possible values when we are returning prematurely.
// This is to ensure that in any circumstance we are not returning empty arrays with wrong values.
func defaultHealResult(storageDisks []StorageAPI, errs []error, bucket, object string) madmin.HealResultItem {
func defaultHealResult(latestXLMeta xlMetaV1, storageDisks []StorageAPI, errs []error, bucket, object string) madmin.HealResultItem {
// Initialize heal result object
result := madmin.HealResultItem{
Type: madmin.HealItemObject,
@@ -545,6 +542,9 @@ func defaultHealResult(storageDisks []StorageAPI, errs []error, bucket, object s
// unable to reliably find the object size.
ObjectSize: -1,
}
if latestXLMeta.IsValid() {
result.ObjectSize = latestXLMeta.Stat.Size
}
for index, disk := range storageDisks {
if disk == nil {
@@ -576,9 +576,14 @@ func defaultHealResult(storageDisks []StorageAPI, errs []error, bucket, object s
})
}
// Default to most common configuration for erasure blocks.
result.ParityBlocks = len(storageDisks) / 2
result.DataBlocks = len(storageDisks) / 2
if !latestXLMeta.IsValid() {
// Default to most common configuration for erasure blocks.
result.ParityBlocks = len(storageDisks) / 2
result.DataBlocks = len(storageDisks) / 2
} else {
result.ParityBlocks = latestXLMeta.Erasure.ParityBlocks
result.DataBlocks = latestXLMeta.Erasure.DataBlocks
}
return result
}
@@ -586,7 +591,7 @@ func defaultHealResult(storageDisks []StorageAPI, errs []error, bucket, object s
// Object is considered dangling/corrupted if any only
// if total disks - a combination of corrupted and missing
// files is lesser than number of data blocks.
func (xl xlObjects) isObjectDangling(metaArr []xlMetaV1, errs []error) (validMeta xlMetaV1, ok bool) {
func isObjectDangling(metaArr []xlMetaV1, errs []error, dataErrs []error) (validMeta xlMetaV1, ok bool) {
// We can consider an object data not reliable
// when xl.json is not found in read quorum disks.
// or when xl.json is not readable in read quorum disks.
@@ -598,6 +603,18 @@ func (xl xlObjects) isObjectDangling(metaArr []xlMetaV1, errs []error) (validMet
corruptedXLJSON++
}
}
var notFoundParts int
for i := range dataErrs {
// Only count part errors, if the error is not
// same as xl.json error. This is to avoid
// double counting when both parts and xl.json
// are not available.
if errs[i] != dataErrs[i] {
if dataErrs[i] == errFileNotFound {
notFoundParts++
}
}
}
for _, m := range metaArr {
if !m.IsValid() {
@@ -612,16 +629,12 @@ func (xl xlObjects) isObjectDangling(metaArr []xlMetaV1, errs []error) (validMet
return validMeta, true
}
// We have valid meta, now verify if we have enough files with data blocks.
return validMeta, (len(xl.getDisks()) - corruptedXLJSON - notFoundXLJSON) < validMeta.Erasure.DataBlocks
// We have valid meta, now verify if we have enough files with parity blocks.
return validMeta, corruptedXLJSON+notFoundXLJSON+notFoundParts > validMeta.Erasure.ParityBlocks
}
// HealObject - heal the given object.
//
// FIXME: If an object object was deleted and one disk was down,
// and later the disk comes back up again, heal on the object
// should delete it.
func (xl xlObjects) HealObject(ctx context.Context, bucket, object string, dryRun bool, remove bool) (hr madmin.HealResultItem, err error) {
// HealObject - heal the given object, automatically deletes the object if stale/corrupted if `remove` is true.
func (xl xlObjects) HealObject(ctx context.Context, bucket, object string, dryRun bool, remove bool, scanMode madmin.HealScanMode) (hr madmin.HealResultItem, err error) {
// Create context that also contains information about the object and bucket.
// The top level handler might not have this information.
reqInfo := logger.GetReqInfo(ctx)
@@ -640,13 +653,12 @@ func (xl xlObjects) HealObject(ctx context.Context, bucket, object string, dryRu
storageDisks := xl.getDisks()
// FIXME: Metadata is read again in the healObject() call below.
// Read metadata files from all the disks
partsMetadata, errs := readAllXLMetadata(healCtx, storageDisks, bucket, object)
// Check if the object is dangling, if yes and user requested
// remove we simply delete it from namespace.
if m, ok := xl.isObjectDangling(partsMetadata, errs); ok {
if m, ok := isObjectDangling(partsMetadata, errs, []error{}); ok {
writeQuorum := m.Erasure.DataBlocks + 1
if m.Erasure.DataBlocks == 0 {
writeQuorum = len(xl.getDisks())/2 + 1
@@ -654,21 +666,46 @@ func (xl xlObjects) HealObject(ctx context.Context, bucket, object string, dryRu
if !dryRun && remove {
err = xl.deleteObject(healCtx, bucket, object, writeQuorum, false)
}
return defaultHealResult(storageDisks, errs, bucket, object), err
return defaultHealResult(xlMetaV1{}, storageDisks, errs, bucket, object), err
}
latestXLMeta, err := getLatestXLMeta(healCtx, partsMetadata, errs)
if err != nil {
return defaultHealResult(storageDisks, errs, bucket, object), toObjectErr(err, bucket, object)
return defaultHealResult(xlMetaV1{}, storageDisks, errs, bucket, object), toObjectErr(err, bucket, object)
}
// Lock the object before healing.
objectLock := xl.nsMutex.NewNSLock(bucket, object)
if lerr := objectLock.GetRLock(globalHealingTimeout); lerr != nil {
return defaultHealResult(storageDisks, errs, bucket, object), lerr
return defaultHealResult(latestXLMeta, storageDisks, errs, bucket, object), lerr
}
defer objectLock.RUnlock()
errCount := 0
for _, err := range errs {
if err != nil {
errCount++
}
}
if errCount == len(errs) {
// Only if we get errors from all the disks we return error. Else we need to
// continue to return filled madmin.HealResultItem struct which includes info
// on what disks the file is available etc.
if reducedErr := reduceReadQuorumErrs(ctx, errs, nil, latestXLMeta.Erasure.DataBlocks); reducedErr != nil {
if m, ok := isObjectDangling(partsMetadata, errs, []error{}); ok {
writeQuorum := m.Erasure.DataBlocks + 1
if m.Erasure.DataBlocks == 0 {
writeQuorum = len(storageDisks)/2 + 1
}
if !dryRun && remove {
err = xl.deleteObject(ctx, bucket, object, writeQuorum, false)
}
}
return defaultHealResult(latestXLMeta, storageDisks, errs, bucket, object), toObjectErr(reducedErr, bucket, object)
}
}
// Heal the object.
return healObject(healCtx, xl.getDisks(), bucket, object, latestXLMeta.Erasure.DataBlocks, dryRun)
return xl.healObject(healCtx, bucket, object, partsMetadata, errs, latestXLMeta, dryRun, remove, scanMode)
}
+89 -2
View File
@@ -21,6 +21,8 @@ import (
"context"
"path/filepath"
"testing"
"github.com/minio/minio/pkg/madmin"
)
// Tests undoes and validates if the undoing completes successfully.
@@ -56,6 +58,91 @@ func TestUndoMakeBucket(t *testing.T) {
}
}
func TestHealObjectCorrupted(t *testing.T) {
nDisks := 16
fsDirs, err := getRandomDisks(nDisks)
if err != nil {
t.Fatal(err)
}
defer removeRoots(fsDirs)
// Everything is fine, should return nil
obj, _, err := initObjectLayer(mustGetNewEndpointList(fsDirs...))
if err != nil {
t.Fatal(err)
}
bucket := "bucket"
object := "object"
data := bytes.Repeat([]byte("a"), 5*1024*1024)
var opts ObjectOptions
err = obj.MakeBucketWithLocation(context.Background(), bucket, "")
if err != nil {
t.Fatalf("Failed to make a bucket - %v", err)
}
// Create an object with multiple parts uploaded in decreasing
// part number.
uploadID, err := obj.NewMultipartUpload(context.Background(), bucket, object, opts)
if err != nil {
t.Fatalf("Failed to create a multipart upload - %v", err)
}
var uploadedParts []CompletePart
for _, partID := range []int{2, 1} {
pInfo, err1 := obj.PutObjectPart(context.Background(), bucket, object, uploadID, partID, mustGetPutObjReader(t, bytes.NewReader(data), int64(len(data)), "", ""), opts)
if err1 != nil {
t.Fatalf("Failed to upload a part - %v", err1)
}
uploadedParts = append(uploadedParts, CompletePart{
PartNumber: pInfo.PartNumber,
ETag: pInfo.ETag,
})
}
_, err = obj.CompleteMultipartUpload(context.Background(), bucket, object, uploadID, uploadedParts, ObjectOptions{})
if err != nil {
t.Fatalf("Failed to complete multipart upload - %v", err)
}
// Remove the object backend files from the first disk.
xl := obj.(*xlObjects)
firstDisk := xl.storageDisks[0]
err = firstDisk.DeleteFile(bucket, filepath.Join(object, xlMetaJSONFile))
if err != nil {
t.Fatalf("Failed to delete a file - %v", err)
}
_, err = obj.HealObject(context.Background(), bucket, object, false, false, madmin.HealNormalScan)
if err != nil {
t.Fatalf("Failed to heal object - %v", err)
}
_, err = firstDisk.StatFile(bucket, filepath.Join(object, xlMetaJSONFile))
if err != nil {
t.Errorf("Expected xl.json file to be present but stat failed - %v", err)
}
// Delete xl.json from more than read quorum number of disks, to create a corrupted situation.
for i := 0; i <= len(xl.storageDisks)/2; i++ {
xl.storageDisks[i].DeleteFile(bucket, filepath.Join(object, xlMetaJSONFile))
}
// Try healing now, expect to receive errDiskNotFound.
_, err = obj.HealObject(context.Background(), bucket, object, false, true, madmin.HealDeepScan)
if err != nil {
t.Errorf("Expected nil but received %v", err)
}
// since majority of xl.jsons are not available, object should be successfully deleted.
_, err = obj.GetObjectInfo(context.Background(), bucket, object, ObjectOptions{})
if _, ok := err.(ObjectNotFound); !ok {
t.Errorf("Expect %v but received %v", ObjectNotFound{Bucket: bucket, Object: object}, err)
}
}
// Tests healing of object.
func TestHealObjectXL(t *testing.T) {
nDisks := 16
@@ -114,7 +201,7 @@ func TestHealObjectXL(t *testing.T) {
t.Fatalf("Failed to delete a file - %v", err)
}
_, err = obj.HealObject(context.Background(), bucket, object, false, false)
_, err = obj.HealObject(context.Background(), bucket, object, false, false, madmin.HealNormalScan)
if err != nil {
t.Fatalf("Failed to heal object - %v", err)
}
@@ -130,7 +217,7 @@ func TestHealObjectXL(t *testing.T) {
}
// Try healing now, expect to receive errDiskNotFound.
_, err = obj.HealObject(context.Background(), bucket, object, false, false)
_, err = obj.HealObject(context.Background(), bucket, object, false, false, madmin.HealDeepScan)
// since majority of xl.jsons are not available, object quorum can't be read properly and error will be errXLReadQuorum
if _, ok := err.(InsufficientReadQuorum); !ok {
t.Errorf("Expected %v but received %v", InsufficientReadQuorum{}, err)
+3 -3
View File
@@ -23,7 +23,7 @@ func (xl xlObjects) ListBucketsHeal(ctx context.Context) ([]BucketInfo, error) {
return nil, nil
}
// This is not implemented/needed anymore, look for xl-sets.ListObjectsHeal()
func (xl xlObjects) ListObjectsHeal(ctx context.Context, bucket, prefix, marker, delimiter string, maxKeys int) (loi ListObjectsInfo, e error) {
return loi, nil
// This is not implemented/needed anymore, look for xl-sets.HealObjects()
func (xl xlObjects) HealObjects(ctx context.Context, bucket, prefix string, healFn func(string, string) error) (e error) {
return nil
}
+2 -3
View File
@@ -69,8 +69,7 @@ func (xl xlObjects) listObjects(ctx context.Context, bucket, prefix, marker, del
recursive = false
}
heal := false // true only for xl.ListObjectsHeal
walkResultCh, endWalkCh := xl.listPool.Release(listParams{bucket, recursive, marker, prefix, heal})
walkResultCh, endWalkCh := xl.listPool.Release(listParams{bucket, recursive, marker, prefix})
if walkResultCh == nil {
endWalkCh = make(chan struct{})
isLeaf := xl.isObject
@@ -125,7 +124,7 @@ func (xl xlObjects) listObjects(ctx context.Context, bucket, prefix, marker, del
}
}
params := listParams{bucket, recursive, nextMarker, prefix, heal}
params := listParams{bucket, recursive, nextMarker, prefix}
if !eof {
xl.listPool.Set(params, walkResultCh, endWalkCh)
}
+1 -1
View File
@@ -624,7 +624,7 @@ func (xl xlObjects) CompleteMultipartUpload(ctx context.Context, bucket string,
// Check if an object is present as one of the parent dir.
// -- FIXME. (needs a new kind of lock).
if xl.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return oi, toObjectErr(errFileAccessDenied, bucket, object)
return oi, toObjectErr(errFileParentIsFile, bucket, object)
}
// Calculate s3 compatible md5sum for complete multipart.
+2 -2
View File
@@ -577,7 +577,7 @@ func (xl xlObjects) putObject(ctx context.Context, bucket string, object string,
// -- FIXME. (needs a new kind of lock).
// -- FIXME (this also causes performance issue when disks are down).
if xl.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
if err = xl.putObjectDir(ctx, minioMetaTmpBucket, tempObj, writeQuorum); err != nil {
@@ -608,7 +608,7 @@ func (xl xlObjects) putObject(ctx context.Context, bucket string, object string,
// -- FIXME. (needs a new kind of lock).
// -- FIXME (this also causes performance issue when disks are down).
if xl.parentDirIsObject(ctx, bucket, path.Dir(object)) {
return ObjectInfo{}, toObjectErr(errFileAccessDenied, bucket, object)
return ObjectInfo{}, toObjectErr(errFileParentIsFile, bucket, object)
}
// Limit the reader to its provided size if specified.
+3 -2
View File
@@ -28,6 +28,7 @@ import (
"time"
humanize "github.com/dustin/go-humanize"
"github.com/minio/minio/pkg/madmin"
)
func TestRepeatPutObjectPart(t *testing.T) {
@@ -308,7 +309,7 @@ func TestHealing(t *testing.T) {
t.Fatal(err)
}
_, err = xl.HealObject(context.Background(), bucket, object, false, false)
_, err = xl.HealObject(context.Background(), bucket, object, false, false, madmin.HealNormalScan)
if err != nil {
t.Fatal(err)
}
@@ -337,7 +338,7 @@ func TestHealing(t *testing.T) {
t.Fatal(err)
}
_, err = xl.HealObject(context.Background(), bucket, object, false, false)
_, err = xl.HealObject(context.Background(), bucket, object, false, false, madmin.HealDeepScan)
if err != nil {
t.Fatal(err)
}
+26 -23
View File
@@ -19,6 +19,7 @@ package main
import (
"bufio"
"crypto/tls"
"errors"
"fmt"
"io/ioutil"
"log"
@@ -45,23 +46,23 @@ func getStartTime() time.Time {
di, err := os.Stat("/proc/1")
if err != nil {
// Cant stat proc dir successfully, exit with error
log.Fatal(err.Error())
log.Fatalln(err)
}
return di.ModTime()
}
// Returns the ip:port of the Minio process
// running in the container
func findEndpoint() string {
func findEndpoint() (string, error) {
cmd := exec.Command("netstat", "-ntlp")
stdout, err := cmd.StdoutPipe()
if err != nil {
// error getting stdout pipe
log.Fatal(err.Error())
return "", err
}
if err = cmd.Start(); err != nil {
// error starting the command
log.Fatal(err.Error())
// error executing the command.
return "", err
}
// split netstat output in rows
scanner := bufio.NewScanner(stdout)
@@ -85,20 +86,21 @@ func findEndpoint() string {
}
// wait for cmd to complete before return
if err = cmd.Wait(); err != nil {
// command failed to run
log.Fatal(err.Error())
return "", err
}
// return joint address and port
return strings.Join([]string{addr, port}, ":")
return strings.Join([]string{addr, port}, ":"), nil
}
}
if err = scanner.Err(); err != nil {
return "", err
}
if err = cmd.Wait(); err != nil {
// command failed to run
log.Fatal(err.Error())
return "", err
}
// minio process not found, exit with error
os.Exit(1)
return ""
return "", errors.New("no minio process found")
}
func main() {
@@ -110,16 +112,16 @@ func main() {
// Refer: https://github.com/moby/moby/pull/28938#issuecomment-301753272
if (time.Now().Sub(startTime) / time.Second) < initGraceTime {
os.Exit(0)
} else {
endPoint := findEndpoint()
if (time.Now().Sub(startTime) / time.Second) > initGraceTime {
endPoint, err := findEndpoint()
if err != nil {
log.Fatalln(err)
}
u, err := url.Parse(fmt.Sprintf("http://%s%s", endPoint, healthPath))
if err != nil {
// Could not parse URL successfully
log.Fatal(err.Error())
log.Fatalln(err)
}
// Minio server may be using self-signed or CA certificates. To avoid
// making Docker setup complicated, we skip verifying certificates here.
// This is because, following request tests for health status within
@@ -132,7 +134,7 @@ func main() {
resp, err := client.Get(u.String())
if err != nil {
// GET failed exit
log.Fatal(err.Error())
log.Fatalln(err)
}
if resp.StatusCode == http.StatusOK {
// Drain any response.
@@ -145,7 +147,7 @@ func main() {
// Drain any response.
xhttp.DrainBody(resp.Body)
// GET failed exit
log.Fatal(err.Error())
log.Fatalln(err)
}
bodyString := string(bodyBytes)
// Drain any response.
@@ -157,7 +159,7 @@ func main() {
resp, err = client.Get(u.String())
if err != nil {
// GET failed exit
log.Fatal(err.Error())
log.Fatalln(err)
}
if resp.StatusCode == http.StatusOK {
// Drain any response.
@@ -168,8 +170,9 @@ func main() {
// Drain any response.
xhttp.DrainBody(resp.Body)
}
// Execution reaching here means none of
// the success cases were satisfied
os.Exit(1)
}
// Execution reaching here means none of
// the success cases were satisfied
os.Exit(1)
os.Exit(0)
}
@@ -5,7 +5,7 @@ version: '2'
# 9001 through 9004.
services:
minio1:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
volumes:
- data1:/data
ports:
@@ -15,7 +15,7 @@ services:
MINIO_SECRET_KEY: minio123
command: server http://minio1/data http://minio2/data http://minio3/data http://minio4/data
minio2:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
volumes:
- data2:/data
ports:
@@ -25,7 +25,7 @@ services:
MINIO_SECRET_KEY: minio123
command: server http://minio1/data http://minio2/data http://minio3/data http://minio4/data
minio3:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
volumes:
- data3:/data
ports:
@@ -35,7 +35,7 @@ services:
MINIO_SECRET_KEY: minio123
command: server http://minio1/data http://minio2/data http://minio3/data http://minio4/data
minio4:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
volumes:
- data4:/data
ports:
@@ -2,7 +2,7 @@ version: '3.1'
services:
minio1:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio1
volumes:
- minio1-data:/export
@@ -24,7 +24,7 @@ services:
- access_key
minio2:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio2
volumes:
- minio2-data:/export
@@ -46,7 +46,7 @@ services:
- access_key
minio3:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio3
volumes:
- minio3-data:/export
@@ -68,7 +68,7 @@ services:
- access_key
minio4:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio4
volumes:
- minio4-data:/export
@@ -2,7 +2,7 @@ version: '3'
services:
minio1:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio1
volumes:
- minio1-data:/export
@@ -24,7 +24,7 @@ services:
command: server http://minio1/export http://minio2/export http://minio3/export http://minio4/export
minio2:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio2
volumes:
- minio2-data:/export
@@ -46,7 +46,7 @@ services:
command: server http://minio1/export http://minio2/export http://minio3/export http://minio4/export
minio3:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio3
volumes:
- minio3-data:/export
@@ -68,7 +68,7 @@ services:
command: server http://minio1/export http://minio2/export http://minio3/export http://minio4/export
minio4:
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
hostname: minio4
volumes:
- minio4-data:/export
+1 -1
View File
@@ -197,7 +197,7 @@ kubectl delete statefulset minio \
If your cluster does not have a storage solution or PV abstraction, you must explicitly define what nodes you wish to run Minio on, and define a homogeneous path to a local fast block device available on every host.
This must be changed in the example daemonset: [minio-distributed-daemonset.yaml](minio-distributed-daemonset.yaml)
This must be changed in the example daemonset: [minio-distributed-daemonset.yaml](https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes/minio-distributed-daemonset.yaml)
Specifically the hostpath:
```yaml
@@ -30,7 +30,7 @@ spec:
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
# Unfortunately you must manually define each server. Perhaps autodiscovery via DNS can be implemented in the future.
args:
- server
@@ -21,7 +21,7 @@ spec:
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
args:
- server
- http://minio-0.minio.default.svc.cluster.local/data
@@ -21,7 +21,7 @@ spec:
containers:
- name: minio
# Pulls the default Minio image from Docker Hub
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
args:
- gateway
- gcs
@@ -29,7 +29,7 @@ spec:
- name: data
mountPath: "/data"
# Pulls the lastest Minio image from Docker Hub
image: minio/minio:RELEASE.2019-02-26T19-51-46Z
image: minio/minio:RELEASE.2019-03-20T22-38-47Z
args:
- server
- /data
+192
View File
@@ -0,0 +1,192 @@
module github.com/minio/minio
require (
cloud.google.com/go v0.26.0
contrib.go.opencensus.io/exporter/ocagent v0.4.7 // indirect
contrib.go.opencensus.io/exporter/stackdriver v0.0.0-20180919222851-d1e19f5c23e9 // indirect
github.com/Azure/azure-sdk-for-go v26.4.0+incompatible
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
github.com/Azure/go-autorest v11.5.2+incompatible
github.com/Microsoft/go-winio v0.4.12 // indirect
github.com/NYTimes/gziphandler v1.1.1 // indirect
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
github.com/SAP/go-hdb v0.14.0 // indirect
github.com/SermoDigital/jose v0.9.1 // indirect
github.com/Shopify/toxiproxy v2.1.4+incompatible // indirect
github.com/alecthomas/participle v0.0.0-20190103085315-bf8340a459bd
github.com/aliyun/alibaba-cloud-sdk-go v0.0.0-20190315122603-6f9e54af456e // indirect
github.com/aliyun/aliyun-oss-go-sdk v0.0.0-20170925032315-6fe16293d6b7
github.com/araddon/gou v0.0.0-20190110011759-c797efecbb61 // indirect
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf // indirect
github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f // indirect
github.com/bcicen/jstream v0.0.0-20190220045926-16c1f8af81c2
github.com/bitly/go-hostpool v0.0.0-20171023180738-a3a6125de932 // indirect
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869 // indirect
github.com/boombuler/barcode v1.0.0 // indirect
github.com/briankassouf/jose v0.9.1 // indirect
github.com/cenkalti/backoff v2.1.1+incompatible // indirect
github.com/centrify/cloud-golang-sdk v0.0.0-20190214225812-119110094d0f // indirect
github.com/cheggaaa/pb v0.0.0-20160713104425-73ae1d68fe0b
github.com/chrismalek/oktasdk-go v0.0.0-20181212195951-3430665dfaa0 // indirect
github.com/containerd/continuity v0.0.0-20181203112020-004b46473808 // indirect
github.com/coredns/coredns v0.0.0-20180121192821-d4bf076ccf4e
github.com/coreos/etcd v0.0.0-20180703215944-e4425ee79f2f
github.com/coreos/go-oidc v2.0.0+incompatible // indirect
github.com/dancannon/gorethink v4.0.0+incompatible // indirect
github.com/denisenkom/go-mssqldb v0.0.0-20190315220205-a8ed825ac853 // indirect
github.com/dgrijalva/jwt-go v0.0.0-20180921172315-3af4c746e1c2
github.com/dimchansky/utfbom v1.1.0 // indirect
github.com/djherbis/atime v0.0.0-20170215084934-89517e96e10b
github.com/dnaeon/go-vcr v1.0.1 // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.3.3 // indirect
github.com/duosecurity/duo_api_golang v0.0.0-20190308151101-6c680f768e74 // indirect
github.com/dustin/go-humanize v0.0.0-20170228161531-259d2a102b87
github.com/eapache/go-resiliency v0.0.0-20160104191539-b86b1ec0dd42 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20160609142408-bb955e01b934 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/eclipse/paho.mqtt.golang v0.0.0-20181129145454-379fd9f99ba5
github.com/elazarl/go-bindata-assetfs v0.0.0-20151224045452-57eb5e1fc594
github.com/fatih/color v1.7.0
github.com/fatih/structs v1.1.0
github.com/fullsailor/pkcs7 v0.0.0-20180613152042-8306686428a5 // indirect
github.com/gammazero/deque v0.0.0-20190130191400-2afb3858e9c7 // indirect
github.com/gammazero/workerpool v0.0.0-20181230203049-86a96b5d5d92 // indirect
github.com/garyburd/redigo v1.6.0 // indirect
github.com/go-errors/errors v1.0.1 // indirect
github.com/go-ldap/ldap v3.0.2+incompatible // indirect
github.com/go-sql-driver/mysql v0.0.0-20181218123637-c45f530f8e7f
github.com/go-stomp/stomp v2.0.2+incompatible // indirect
github.com/go-test/deep v1.0.1 // indirect
github.com/gocql/gocql v0.0.0-20190301043612-f6df8288f9b4 // indirect
github.com/gogo/protobuf v1.2.1 // indirect
github.com/golang/snappy v0.0.1
github.com/gomodule/redigo v0.0.0-20190205135352-43fe51054af5
github.com/google/go-github v17.0.0+incompatible // indirect
github.com/google/go-querystring v1.0.0 // indirect
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf // indirect
github.com/google/martian v2.1.0+incompatible // indirect
github.com/gopherjs/gopherjs v0.0.0-20190309154008-847fc94819f9 // indirect
github.com/gorhill/cronexpr v0.0.0-20180427100037-88b0669f7d75 // indirect
github.com/gorilla/context v0.0.0-20160525203319-aed02d124ae4 // indirect
github.com/gorilla/handlers v0.0.0-20160410185317-66e6c6f01d8d
github.com/gorilla/mux v0.0.0-20160605233521-9fa818a44c2b
github.com/gorilla/rpc v0.0.0-20160517062331-bd3317b8f670
github.com/gotestyourself/gotestyourself v2.2.0+incompatible // indirect
github.com/hashicorp/consul v1.4.3 // indirect
github.com/hashicorp/go-cleanhttp v0.0.0-20171218145408-d5fe4b57a186 // indirect
github.com/hashicorp/go-gcp-common v0.0.0-20180425173946-763e39302965 // indirect
github.com/hashicorp/go-hclog v0.8.0 // indirect
github.com/hashicorp/go-memdb v0.0.0-20190306140544-eea0b16292ad // indirect
github.com/hashicorp/go-plugin v0.0.0-20190220160451-3f118e8ee104 // indirect
github.com/hashicorp/go-retryablehttp v0.0.0-20180718195005-e651d75abec6 // indirect
github.com/hashicorp/go-rootcerts v0.0.0-20160503143440-6bb64b370b90 // indirect
github.com/hashicorp/go-sockaddr v1.0.2 // indirect
github.com/hashicorp/go-version v0.0.0-20160214002439-2e7f5ea8e27b
github.com/hashicorp/golang-lru v0.5.1 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/hashicorp/nomad v0.8.7 // indirect
github.com/hashicorp/raft v1.0.0 // indirect
github.com/hashicorp/serf v0.8.2 // indirect
github.com/hashicorp/vault v0.0.0-20181121181053-d4367e581fe1
github.com/hashicorp/vault-plugin-auth-alicloud v0.0.0-20190311155555-98628998247d // indirect
github.com/hashicorp/vault-plugin-auth-azure v0.0.0-20190201222632-0af1d040b5b3 // indirect
github.com/hashicorp/vault-plugin-auth-centrify v0.0.0-20180816201131-66b0a34a58bf // indirect
github.com/hashicorp/vault-plugin-auth-gcp v0.0.0-20190201215414-7d4c2101e7d0 // indirect
github.com/hashicorp/vault-plugin-auth-jwt v0.0.0-20190314211503-86b44673ce1e // indirect
github.com/hashicorp/vault-plugin-auth-kubernetes v0.0.0-20190201222209-db96aa4ab438 // indirect
github.com/hashicorp/vault-plugin-secrets-ad v0.0.0-20190131222416-4796d9980125 // indirect
github.com/hashicorp/vault-plugin-secrets-alicloud v0.0.0-20190131211812-b0abe36195cb // indirect
github.com/hashicorp/vault-plugin-secrets-azure v0.0.0-20181207232500-0087bdef705a // indirect
github.com/hashicorp/vault-plugin-secrets-gcp v0.0.0-20190311200649-621231cb86fe // indirect
github.com/hashicorp/vault-plugin-secrets-gcpkms v0.0.0-20190116164938-d6b25b0b4a39 // indirect
github.com/hashicorp/vault-plugin-secrets-kv v0.0.0-20190315192709-dccffee64925 // indirect
github.com/inconshreveable/go-update v0.0.0-20160112193335-8152e7eb6ccf
github.com/jeffchao/backoff v0.0.0-20140404060208-9d7fd7aa17f2 // indirect
github.com/jefferai/jsonx v1.0.0 // indirect
github.com/json-iterator/go v1.1.6 // indirect
github.com/keybase/go-crypto v0.0.0-20190312101036-b475f2ecc1fe // indirect
github.com/klauspost/compress v1.3.0 // indirect
github.com/klauspost/cpuid v0.0.0-20160106104451-349c67577817 // indirect
github.com/klauspost/crc32 v0.0.0-20161016154125-cb6bfca970f6 // indirect
github.com/klauspost/pgzip v0.0.0-20180606150939-90b2c57fba35
github.com/klauspost/reedsolomon v0.0.0-20190210214925-2b210cf0866d
github.com/lib/pq v0.0.0-20181016162627-9eb73efc1fcc
github.com/marstr/guid v1.1.0 // indirect
github.com/mattbaird/elastigo v0.0.0-20170123220020-2fe47fd29e4b // indirect
github.com/mattn/go-isatty v0.0.4
github.com/michaelklishin/rabbit-hole v1.5.0 // indirect
github.com/miekg/dns v1.0.14
github.com/minio/blazer v0.0.0-20171126203752-2081f5bf0465
github.com/minio/cli v0.0.0-20170227073228-b8ae5507c0ce
github.com/minio/dsync v0.0.0-20190104003057-61c41ffdeea2
github.com/minio/highwayhash v0.0.0-20181220011308-93ed73d64169
github.com/minio/lsync v0.0.0-20190207022115-a4e43e3d0887
github.com/minio/mc v0.0.0-20190311071728-2e612b23d665
github.com/minio/minio-go v0.0.0-20190227180923-59af836a7e6d
github.com/minio/parquet-go v0.0.0-20190318185229-9d767baf1679
github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16
github.com/minio/sio v0.0.0-20180327104954-6a41828a60f0
github.com/mitchellh/copystructure v1.0.0 // indirect
github.com/mitchellh/go-homedir v1.1.0
github.com/mitchellh/go-testing-interface v1.0.0 // indirect
github.com/mitchellh/hashstructure v1.0.0 // indirect
github.com/mitchellh/pointerstructure v0.0.0-20170205204203-f2329fcfa9e2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.1 // indirect
github.com/nats-io/gnatsd v1.4.1 // indirect
github.com/nats-io/go-nats v0.0.0-20161120202126-6b6bf392d34d // indirect
github.com/nats-io/go-nats-streaming v0.0.0-20161216191029-077898146bfb
github.com/nats-io/nats v0.0.0-20160916181735-70b70be17b77
github.com/nats-io/nats-streaming-server v0.12.2 // indirect
github.com/nats-io/nuid v1.0.0 // indirect
github.com/nsqio/go-nsq v0.0.0-20181028195256-0527e80f3ba5
github.com/onsi/ginkgo v1.8.0 // indirect
github.com/onsi/gomega v1.5.0 // indirect
github.com/opencontainers/go-digest v1.0.0-rc1 // indirect
github.com/opencontainers/image-spec v1.0.1 // indirect
github.com/opencontainers/runc v0.1.1 // indirect
github.com/ory-am/common v0.4.0 // indirect
github.com/ory/dockertest v3.3.4+incompatible // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pborman/uuid v1.2.0 // indirect
github.com/pkg/profile v1.2.1
github.com/pquerna/cachecontrol v0.0.0-20180517163645-1555304b9b35 // indirect
github.com/pquerna/otp v1.1.0 // indirect
github.com/prometheus/client_golang v0.9.2
github.com/rjeczalik/notify v0.9.2
github.com/rs/cors v0.0.0-20190116175910-76f58f330d76
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/samuel/go-zookeeper v0.0.0-20180130194729-c4fab1ac1bec // indirect
github.com/satori/go.uuid v1.2.0 // indirect
github.com/segmentio/go-prompt v0.0.0-20161017233205-f0d19b6901ad
github.com/skyrings/skyring-common v0.0.0-20160324141443-762fd2bfc12e
github.com/smartystreets/goconvey v0.0.0-20190306220146-200a235640ff // indirect
github.com/streadway/amqp v0.0.0-20160311215503-2e25825abdbd
github.com/tidwall/gjson v1.1.4
github.com/tidwall/sjson v1.0.4
github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a
go.uber.org/atomic v1.3.2
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3 // indirect
golang.org/x/net v0.0.0-20190318221613-d196dffd7c2b
golang.org/x/sys v0.0.0-20190318195719-6c81ef8f67ca
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c
golang.org/x/tools v0.0.0-20190318200714-bb1270c20edf // indirect
google.golang.org/api v0.0.0-20180916000451-19ff8768a5c0
google.golang.org/genproto v0.0.0-20180918203901-c3f76f3b92d1 // indirect
gopkg.in/Shopify/sarama.v1 v1.10.1
gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d // indirect
gopkg.in/gorethink/gorethink.v4 v4.1.0 // indirect
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce // indirect
gopkg.in/olivere/elastic.v5 v5.0.31
gopkg.in/ory-am/dockertest.v2 v2.2.3 // indirect
gopkg.in/square/go-jose.v2 v2.3.0 // indirect
gopkg.in/yaml.v2 v2.2.2
gotest.tools v2.2.0+incompatible // indirect
k8s.io/api v0.0.0-20190313115550-3c12c96769cc // indirect
k8s.io/apimachinery v0.0.0-20190313115320-c9defaaddf6f // indirect
k8s.io/klog v0.2.0 // indirect
layeh.com/radius v0.0.0-20190118135028-0f678f039617 // indirect
sigs.k8s.io/yaml v1.1.0 // indirect
)
+673
View File
@@ -0,0 +1,673 @@
cloud.google.com/go v0.23.0 h1:w1svupRqvZnfjN9+KksMiggoIRQuMzWkVzpxcR96xDs=
cloud.google.com/go v0.23.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.26.0 h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
contrib.go.opencensus.io/exporter/ocagent v0.4.7 h1:7NiGV38nUxXevUGX5rJdG6QBRRXRDRb6ABwdYHjhiXI=
contrib.go.opencensus.io/exporter/ocagent v0.4.7/go.mod h1:+KkYrcvvEN0E5ls626sqMv8PdMx2931feKtzIwP01qI=
contrib.go.opencensus.io/exporter/stackdriver v0.0.0-20180919222851-d1e19f5c23e9 h1:qGDL/as+reGch2TnQl053h96loeNRKASdiRqnKQDLko=
contrib.go.opencensus.io/exporter/stackdriver v0.0.0-20180919222851-d1e19f5c23e9/go.mod h1:hNe5qQofPbg6bLQY5wHCvQ7o+2E5P8PkegEuQ+MyRw0=
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
git.apache.org/thrift.git v0.12.0 h1:CMxsZlAmxKs+VAZMlDDL0wXciMblJcutQbEe3A9CYUM=
git.apache.org/thrift.git v0.12.0/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
github.com/Azure/azure-sdk-for-go v26.4.0+incompatible h1:ISw3xYFYPGBmcwP7CQjzQDoYhkywcIVfYzo4CHgQzOw=
github.com/Azure/azure-sdk-for-go v26.4.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
github.com/Azure/go-autorest v11.5.2+incompatible h1:NTIEargbhAGNWuT7QEXJ2fqLMFvatupHIscb9FYwVOg=
github.com/Azure/go-autorest v11.5.2+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
github.com/Jeffail/gabs v1.1.1 h1:V0uzR08Hj22EX8+8QMhyI9sX2hwRu+/RJhJUmnwda/E=
github.com/Jeffail/gabs v1.1.1/go.mod h1:6xMvQMK4k33lb7GUUpaAPh6nKMmemQeg5d4gn7/bOXc=
github.com/Microsoft/go-winio v0.4.12 h1:xAfWHN1IrQ0NJ9TBC0KBZoqLjzDTr1ML+4MywiUOryc=
github.com/Microsoft/go-winio v0.4.12/go.mod h1:VhR8bwka0BXejwEJY73c50VrPtXAaKcyvVC4A4RozmA=
github.com/NYTimes/gziphandler v1.1.1 h1:ZUDjpQae29j0ryrS0u/B8HZfJBtBQHjqw2rQ2cqUQ3I=
github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMomdKFjzJNB0c=
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 h1:TngWCqHvy9oXAN6lEVMRuU21PR1EtLVZJmdB18Gu3Rw=
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5/go.mod h1:lmUJ/7eu/Q8D7ML55dXQrVaamCz2vxCfdQBasLZfHKk=
github.com/SAP/go-hdb v0.14.0 h1:e6lvHlwRcjjDGeiv8PtKZHadTWsEGVMr9Zqc/iolwgM=
github.com/SAP/go-hdb v0.14.0/go.mod h1:7fdQLVC2lER3urZLjZCm0AuMQfApof92n3aylBPEkMo=
github.com/SermoDigital/jose v0.9.1 h1:atYaHPD3lPICcbK1owly3aPm0iaJGSGPi0WD4vLznv8=
github.com/SermoDigital/jose v0.9.1/go.mod h1:ARgCUhI1MHQH+ONky/PAtmVHQrP5JlGY0F3poXOp/fA=
github.com/Shopify/toxiproxy v2.1.4+incompatible h1:TKdv8HiTLgE5wdJuEML90aBgNWsokNbMijUGhmcoBJc=
github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI=
github.com/alecthomas/participle v0.0.0-20190103085315-bf8340a459bd h1:Fx8zE0HVKjLkrRpyds1EK9HJeXWTVlO2dqWC5UNR4Wc=
github.com/alecthomas/participle v0.0.0-20190103085315-bf8340a459bd/go.mod h1:SW6HZGeZgSIpcUWX3fXpfZhuaWHnmoD5KCVaqSaNTkk=
github.com/aliyun/alibaba-cloud-sdk-go v0.0.0-20190315122603-6f9e54af456e h1:YzotKd+gUHEK6v3WEVQUT/UbZdqJJ4Nlqtk/xyhd/q4=
github.com/aliyun/alibaba-cloud-sdk-go v0.0.0-20190315122603-6f9e54af456e/go.mod h1:T9M45xf79ahXVelWoOBmH0y4aC1t5kXO5BxwyakgIGA=
github.com/aliyun/aliyun-oss-go-sdk v0.0.0-20170925032315-6fe16293d6b7 h1:uHceyjNJ6fFv1Vv2zOrB7Qm8VnVjQ/TuJ8mwV7pHPKk=
github.com/aliyun/aliyun-oss-go-sdk v0.0.0-20170925032315-6fe16293d6b7/go.mod h1:T/Aws4fEfogEE9v+HPhhw+CntffsBHJ8nXQCwKr0/g8=
github.com/araddon/gou v0.0.0-20190110011759-c797efecbb61 h1:Xz25cuW4REGC5W5UtpMU3QItMIImag615HiQcRbxqKQ=
github.com/araddon/gou v0.0.0-20190110011759-c797efecbb61/go.mod h1:ikc1XA58M+Rx7SEbf0bLJCfBkwayZ8T5jBo5FXK8Uz8=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da h1:8GUt8eRujhVEGZFFEjBj46YV4rDjvGrNxb0KMWYkL2I=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310 h1:BUAU3CGlLvorLI26FmByPp2eC2qla6E1Tw+scpcg/to=
github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf h1:eg0MeVzsP1G42dRafH3vf+al2vQIJU0YHX+1Tw87oco=
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/aws/aws-sdk-go v1.15.31 h1:ExgD8W8QDeD8Y4CPVlcP/laumxvikDbkVWB+VCHgXxA=
github.com/aws/aws-sdk-go v1.15.31/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0=
github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f h1:ZNv7On9kyUzm7fvRZumSyy/IUiSC7AzL0I1jKKtwooA=
github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f/go.mod h1:AuiFmCCPBSrqvVMvuqFuk0qogytodnVFVSN5CeJB8Gc=
github.com/bcicen/jstream v0.0.0-20190220045926-16c1f8af81c2 h1:M+TYzBcNIRyzPRg66ndEqUMd7oWDmhvdQmaPC6EZNwM=
github.com/bcicen/jstream v0.0.0-20190220045926-16c1f8af81c2/go.mod h1:RDu/qcrnpEdJC/p8tx34+YBFqqX71lB7dOX9QE+ZC4M=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
github.com/bitly/go-hostpool v0.0.0-20171023180738-a3a6125de932 h1:mXoPYz/Ul5HYEDvkta6I8/rnYM5gSdSV2tJ6XbZuEtY=
github.com/bitly/go-hostpool v0.0.0-20171023180738-a3a6125de932/go.mod h1:NOuUCSz6Q9T7+igc/hlvDOUdtWKryOrtFyIVABv/p7k=
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869 h1:DDGfHa7BWjL4YnC6+E63dPcxHo2sUxDIu8g3QgEJdRY=
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
github.com/boombuler/barcode v1.0.0 h1:s1TvRnXwL2xJRaccrdcBQMZxq6X7DvsMogtmJeHDdrc=
github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/briankassouf/jose v0.9.1 h1:HMeR+qpz2PBP7RgR+64nJPA9qcZI74tdkBv5NyD0Mzk=
github.com/briankassouf/jose v0.9.1/go.mod h1:HQhVmdUf7dBNwIIdBTivnCDxcf6IZY3/zrb+uKSJz6Y=
github.com/cenkalti/backoff v2.0.0+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
github.com/cenkalti/backoff v2.1.1+incompatible h1:tKJnvO2kl0zmb/jA5UKAt4VoEVw1qxKWjE/Bpp46npY=
github.com/cenkalti/backoff v2.1.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
github.com/census-instrumentation/opencensus-proto v0.2.0 h1:LzQXZOgg4CQfE6bFvXGM30YZL1WW/M337pXml+GrcZ4=
github.com/census-instrumentation/opencensus-proto v0.2.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/centrify/cloud-golang-sdk v0.0.0-20190214225812-119110094d0f h1:gJzxrodnNd/CtPXjO3WYiakyNzHg3rtAi7rO74ejHYU=
github.com/centrify/cloud-golang-sdk v0.0.0-20190214225812-119110094d0f/go.mod h1:C0rtzmGXgN78pYR0tGJFhtHgkbAs0lIbHwkB81VxDQE=
github.com/cheggaaa/pb v0.0.0-20160713104425-73ae1d68fe0b h1:byvZ+gwmaExJc7n2oYzpN3VCnUDOUwqzo/JrunZmvGk=
github.com/cheggaaa/pb v0.0.0-20160713104425-73ae1d68fe0b/go.mod h1:pQciLPpbU0oxA0h+VJYYLxO+XeDQb5pZijXscXHm81s=
github.com/chrismalek/oktasdk-go v0.0.0-20181212195951-3430665dfaa0 h1:CWU8piLyqoi9qXEUwzOh5KFKGgmSU5ZhktJyYcq6ryQ=
github.com/chrismalek/oktasdk-go v0.0.0-20181212195951-3430665dfaa0/go.mod h1:5d8DqS60xkj9k3aXfL3+mXBH0DPYO0FQjcKosxl+b/Q=
github.com/client9/misspell v0.3.4 h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/containerd/continuity v0.0.0-20181203112020-004b46473808 h1:4BX8f882bXEDKfWIf0wa8HRvpnBoPszJJXL+TVbBw4M=
github.com/containerd/continuity v0.0.0-20181203112020-004b46473808/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/coredns/coredns v0.0.0-20180121192821-d4bf076ccf4e h1:AExHcQe3cJZQk3wgIs9UmW0lR9Bp1hk+X8YK5Bbn6/4=
github.com/coredns/coredns v0.0.0-20180121192821-d4bf076ccf4e/go.mod h1:zASH/MVDgR6XZTbxvOnsZfffS+31vg6Ackf/wo1+AM0=
github.com/coreos/bbolt v1.3.2 h1:wZwiHHUieZCquLkDL0B8UhzreNWsPHooDAG3q34zk0s=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
github.com/coreos/etcd v0.0.0-20180622175804-8f6348a97d48/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/etcd v0.0.0-20180703215944-e4425ee79f2f h1:YcMpzEkbh12yE3+PngDUFe69GCbqvNwBiUJJQZV7vGY=
github.com/coreos/etcd v0.0.0-20180703215944-e4425ee79f2f/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-oidc v2.0.0+incompatible h1:+RStIopZ8wooMx+Vs5Bt8zMXxV1ABl5LbakNExNmZIg=
github.com/coreos/go-oidc v2.0.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
github.com/coreos/go-semver v0.0.0-20180108230905-e214231b295a h1:WqY2Kv7eI1jeoU3pC05YYK/kK4tdXyLzzaBzCR51r9M=
github.com/coreos/go-semver v0.0.0-20180108230905-e214231b295a/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20181031085051-9002847aa142 h1:3jFq2xL4ZajGK4aZY8jz+DAF0FHjI51BXjjSwCzS1Dk=
github.com/coreos/go-systemd v0.0.0-20181031085051-9002847aa142/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbpBpLoyyu8B6e44T7hJy6potg=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/dancannon/gorethink v4.0.0+incompatible h1:KFV7Gha3AuqT+gr0B/eKvGhbjmUv0qGF43aKCIKVE9A=
github.com/dancannon/gorethink v4.0.0+incompatible/go.mod h1:BLvkat9KmZc1efyYwhz3WnybhRZtgF1K929FD8z1avU=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/denisenkom/go-mssqldb v0.0.0-20190315220205-a8ed825ac853 h1:tTngnoO/B6HQnJ+pK8tN7kEAhmhIfaJOutqq/A4/JTM=
github.com/denisenkom/go-mssqldb v0.0.0-20190315220205-a8ed825ac853/go.mod h1:xN/JuLBIz4bjkxNmByTiV1IbhfnYb6oo99phBn4Eqhc=
github.com/dgrijalva/jwt-go v0.0.0-20180921172315-3af4c746e1c2 h1:xhptajUY6xeFJmfsrpVPxXxH0i2JxobVCBDn0iShbkU=
github.com/dgrijalva/jwt-go v0.0.0-20180921172315-3af4c746e1c2/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dimchansky/utfbom v1.1.0 h1:FcM3g+nofKgUteL8dm/UpdRXNC9KmADgTpLKsu0TRo4=
github.com/dimchansky/utfbom v1.1.0/go.mod h1:rO41eb7gLfo8SF1jd9F8HplJm1Fewwi4mQvIirEdv+8=
github.com/djherbis/atime v0.0.0-20170215084934-89517e96e10b h1:e0AH6GKm2gddyuEilb3+K3LmRUb2dK23JDDrZJ7dmD8=
github.com/djherbis/atime v0.0.0-20170215084934-89517e96e10b/go.mod h1:5W+KBIuTwVGcqjIfaTwt+KSYX1o6uep8dtevevQP/f8=
github.com/dnaeon/go-vcr v1.0.1 h1:r8L/HqC0Hje5AXMu1ooW8oyQyOFv4GxqpL0nRP7SLLY=
github.com/dnaeon/go-vcr v1.0.1/go.mod h1:aBB1+wY4s93YsC3HHjMBMrwTj2R9FHDzUr9KyGc8n1E=
github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ=
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
github.com/docker/go-units v0.3.3 h1:Xk8S3Xj5sLGlG5g67hJmYMmUgXv5N4PhkjJHHqrwnTk=
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/duosecurity/duo_api_golang v0.0.0-20190308151101-6c680f768e74 h1:2MIhn2R6oXQbgW5yHfS+d6YqyMfXiu2L55rFZC4UD/M=
github.com/duosecurity/duo_api_golang v0.0.0-20190308151101-6c680f768e74/go.mod h1:UqXY1lYT/ERa4OEAywUqdok1T4RCRdArkhic1Opuavo=
github.com/dustin/go-humanize v0.0.0-20150809201405-1c212aae1d02/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/dustin/go-humanize v0.0.0-20170228161531-259d2a102b87 h1:uPzP/9GIqYKvZAmz4IayKMMZiWRWNtGynUREBtTXPXA=
github.com/dustin/go-humanize v0.0.0-20170228161531-259d2a102b87/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/eapache/go-resiliency v0.0.0-20160104191539-b86b1ec0dd42 h1:f8ERmXYuaC+kCSv2w+y3rBK/oVu6If4DEm3jywJJ0hc=
github.com/eapache/go-resiliency v0.0.0-20160104191539-b86b1ec0dd42/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs=
github.com/eapache/go-xerial-snappy v0.0.0-20160609142408-bb955e01b934 h1:oGLoaVIefp3tiOgi7+KInR/nNPvEpPM6GFo+El7fd14=
github.com/eapache/go-xerial-snappy v0.0.0-20160609142408-bb955e01b934/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/eclipse/paho.mqtt.golang v0.0.0-20181129145454-379fd9f99ba5 h1:2AEPnz/cyD83iOHC1HaW06xSW6/0xCpKkuGEBHvWvYs=
github.com/eclipse/paho.mqtt.golang v0.0.0-20181129145454-379fd9f99ba5/go.mod h1:H9keYFcgq3Qr5OUJm/JZI/i6U7joQ8SYLhZwfeOo6Ts=
github.com/elazarl/go-bindata-assetfs v0.0.0-20151224045452-57eb5e1fc594 h1:McZ/pt/pP/XAbLMDQGzm/iQUwW6OXmKVbFtmH9klWmc=
github.com/elazarl/go-bindata-assetfs v0.0.0-20151224045452-57eb5e1fc594/go.mod h1:v+YaWX3bdea5J/mo8dSETolEo7R71Vk1u8bnjau5yw4=
github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/fatih/structs v0.0.0-20181010231757-878a968ab225/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fullsailor/pkcs7 v0.0.0-20180613152042-8306686428a5 h1:v+vxrd9XS8uWIXG2RK0BHCnXc30qLVQXVqbK+IOmpXk=
github.com/fullsailor/pkcs7 v0.0.0-20180613152042-8306686428a5/go.mod h1:KnogPXtdwXqoenmZCw6S+25EAm2MkxbG0deNDu4cbSA=
github.com/gammazero/deque v0.0.0-20190130191400-2afb3858e9c7 h1:D2LrfOPgGHQprIxmsTpxtzhpmF66HoM6rXSmcqaX7h8=
github.com/gammazero/deque v0.0.0-20190130191400-2afb3858e9c7/go.mod h1:GeIq9qoE43YdGnDXURnmKTnGg15pQz4mYkXSTChbneI=
github.com/gammazero/workerpool v0.0.0-20181230203049-86a96b5d5d92 h1:EipXK6U05IQ2wtuFRn4k3h0+2lXypzItoXGVyf4r9Io=
github.com/gammazero/workerpool v0.0.0-20181230203049-86a96b5d5d92/go.mod h1:w9RqFVO2BM3xwWEcAB8Fwp0OviTBBEiRmSBDfbXnd3w=
github.com/garyburd/redigo v1.6.0 h1:0VruCpn7yAIIu7pWVClQC8wxCJEcG3nyzpMSHKi1PQc=
github.com/garyburd/redigo v1.6.0/go.mod h1:NR3MbYisc3/PwhQ00EMzDiPmrwpPxAn5GI05/YaO1SY=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-errors/errors v1.0.1 h1:LUHzmkK3GUKUrL/1gfBUxAHzcev3apQlezX/+O7ma6w=
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-ini/ini v1.25.4/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
github.com/go-ini/ini v1.42.0 h1:TWr1wGj35+UiWHlBA8er89seFXxzwFn11spilrrj+38=
github.com/go-ini/ini v1.42.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
github.com/go-ldap/ldap v3.0.2+incompatible h1:kD5HQcAzlQ7yrhfn+h+MSABeAy/jAJhvIJ/QDllP44g=
github.com/go-ldap/ldap v3.0.2+incompatible/go.mod h1:qfd9rJvER9Q0/D/Sqn1DfHRoBp40uXYvFoEVrNEPqRc=
github.com/go-sql-driver/mysql v0.0.0-20181218123637-c45f530f8e7f h1:fbIzwEaXt5b2bl9mm+PIufKTSGKk6ZuwSSTQ7iZj7Lo=
github.com/go-sql-driver/mysql v0.0.0-20181218123637-c45f530f8e7f/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
github.com/go-stomp/stomp v2.0.2+incompatible h1:0yPknMJh32lE2xiCFGW5t/KgamhUC4OgCv10wIjx5aw=
github.com/go-stomp/stomp v2.0.2+incompatible/go.mod h1:VqCtqNZv1226A1/79yh+rMiFUcfY3R109np+7ke4n0c=
github.com/go-test/deep v1.0.1 h1:UQhStjbkDClarlmv0am7OXXO4/GaPdCGiUiMTvi28sg=
github.com/go-test/deep v1.0.1/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
github.com/gocql/gocql v0.0.0-20190301043612-f6df8288f9b4 h1:vF83LI8tAakwEwvWZtrIEx7pOySacl2TOxx6eXk4ePo=
github.com/gocql/gocql v0.0.0-20190301043612-f6df8288f9b4/go.mod h1:4Fw1eo5iaEhDUs8XyuhSVCVy52Jq3L+/3GJgYkwc+/0=
github.com/gogo/protobuf v1.2.0 h1:xU6/SpYbvkNYiptHJYEDRseDLvYE7wSqhYYNy0QSUzI=
github.com/gogo/protobuf v1.2.0/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE=
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef h1:veQD95Isof8w9/WXiA+pa3tz3fJXkt5B7QaRBrM62gk=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.1.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.0 h1:kbxbvI4Un1LUWKxufD+BiE6AEExYYgkQLQmLFqA1LFk=
github.com/golang/protobuf v1.3.0/go.mod h1:Qd/q+1AKNOZr9uGQzbzCmRO6sUih6GTPZv6a1/R87v0=
github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/gomodule/redigo v0.0.0-20190205135352-43fe51054af5 h1:HVDzcZHqJUlNfCqPdTMidgAPVeDZR32Vuam64ikElJA=
github.com/gomodule/redigo v0.0.0-20190205135352-43fe51054af5/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c h1:964Od4U6p2jUkFxvCydnIczKteheJEzHRToSGK3Bnlw=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-github v17.0.0+incompatible h1:N0LgJ1j65A7kfXrZnUDaYCs/Sf4rEjNlfyDHW9dolSY=
github.com/google/go-github v17.0.0+incompatible/go.mod h1:zLgOLi98H3fifZn+44m+umXrS52loVEgC2AApnigrVQ=
github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk=
github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf h1:+RRA9JqSOZFfKrOeqr2z77+8R2RKyh8PG66dcu1V0ck=
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/uuid v1.0.0 h1:b4Gk+7WdP/d3HZH8EJsZpvV7EtDOgaZLtnaNGIu1adA=
github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go v2.0.0+incompatible h1:j0GKcs05QVmm7yesiZq2+9cxHkNK9YM6zKx4D2qucQU=
github.com/googleapis/gax-go v2.0.0+incompatible/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gopherjs/gopherjs v0.0.0-20181103185306-d547d1d9531e/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gopherjs/gopherjs v0.0.0-20190309154008-847fc94819f9 h1:Z0f701LpR4dqO92bP6TnIe3ZURClzJtBhds8R8u1HBE=
github.com/gopherjs/gopherjs v0.0.0-20190309154008-847fc94819f9/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorhill/cronexpr v0.0.0-20180427100037-88b0669f7d75 h1:f0n1xnMSmBLzVfsMMvriDyA75NB/oBgILX2GcHXIQzY=
github.com/gorhill/cronexpr v0.0.0-20180427100037-88b0669f7d75/go.mod h1:g2644b03hfBX9Ov0ZBDgXXens4rxSxmqFBbhvKv2yVA=
github.com/gorilla/context v0.0.0-20160525203319-aed02d124ae4 h1:3nOfQt8sRPYbXORD5tJ8YyQ3HlL2Jt3LJ2U17CbNh6I=
github.com/gorilla/context v0.0.0-20160525203319-aed02d124ae4/go.mod h1:kBGZzfjB9CEq2AlWe17Uuf7NDRt0dE0s8S51q0aT7Yg=
github.com/gorilla/handlers v0.0.0-20160410185317-66e6c6f01d8d h1:wroUBCGyWwKzb/MvvcfuO91P7c0cR8oNoFVvNd3B2kU=
github.com/gorilla/handlers v0.0.0-20160410185317-66e6c6f01d8d/go.mod h1:Qkdc/uu4tH4g6mTK6auzZ766c4CA0Ng8+o/OAirnOIQ=
github.com/gorilla/mux v0.0.0-20160605233521-9fa818a44c2b h1:OFvZV3a+25cGJH9dETHw0nk0wV6hLZI7IJijOkXEFS0=
github.com/gorilla/mux v0.0.0-20160605233521-9fa818a44c2b/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
github.com/gorilla/rpc v0.0.0-20160517062331-bd3317b8f670 h1:YzxOFT0J9DX8G4WfjA0j5A9Pih7ED5ztSE/zU/TnBK8=
github.com/gorilla/rpc v0.0.0-20160517062331-bd3317b8f670/go.mod h1:V4h9r+4sF5HnzqbwIez0fKSpANP0zlYd3qR7p36jkTQ=
github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/gotestyourself/gotestyourself v2.2.0+incompatible h1:AQwinXlbQR2HvPjQZOmDhRqsv5mZf+Jb1RnSLxcqZcI=
github.com/gotestyourself/gotestyourself v2.2.0+incompatible/go.mod h1:zZKM6oeNM8k+FRljX1mnzVYeS8wiGgQyvST1/GafPbY=
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0 h1:Iju5GlWwrvL6UBg4zJJt3btmonfrMlCDdsejg4CZE7c=
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.5.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
github.com/grpc-ecosystem/grpc-gateway v1.7.0 h1:tPFY/SM+d656aSgLWO2Eckc3ExwpwwybwdN5Ph20h1A=
github.com/grpc-ecosystem/grpc-gateway v1.7.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed h1:5upAirOpQc1Q53c0bnx2ufif5kANL7bfZWcc6VJWJd8=
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed/go.mod h1:tMWxXQ9wFIaZeTI9F+hmhFiGpFmhOHzyShyFUhRm0H4=
github.com/hashicorp/consul v1.4.3 h1:mNHondWmijBVwPKK94hlkUOgIwXiMQLrnBMsQdKrhDE=
github.com/hashicorp/consul v1.4.3/go.mod h1:mFrjN1mfidgJfYP1xrJCF+AfRhr6Eaqhb2+sfyn/OOI=
github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-cleanhttp v0.0.0-20171218145408-d5fe4b57a186 h1:URgjUo+bs1KwatoNbwG0uCO4dHN4r1jsp4a5AGgHRjo=
github.com/hashicorp/go-cleanhttp v0.0.0-20171218145408-d5fe4b57a186/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-gcp-common v0.0.0-20180425173946-763e39302965 h1:CRWAJ7atEOq9OfAa5rBKvL4wFNbUrPil8hN1knLuEG0=
github.com/hashicorp/go-gcp-common v0.0.0-20180425173946-763e39302965/go.mod h1:LNbios2fdMAuLA1dsYUvUcoCYIfywcCEK8/ooaWjoOA=
github.com/hashicorp/go-hclog v0.0.0-20180709165350-ff2cf002a8dd/go.mod h1:9bjs9uLqI8l75knNv3lV1kA55veR+WUPSiKIWcQHudI=
github.com/hashicorp/go-hclog v0.8.0 h1:z3ollgGRg8RjfJH6UVBaG54R70GFd++QOkvnJH3VSBY=
github.com/hashicorp/go-hclog v0.8.0/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
github.com/hashicorp/go-immutable-radix v1.0.0 h1:AKDB1HM5PWEA7i4nhcpwOrO2byshxBjXVn/J/3+z5/0=
github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
github.com/hashicorp/go-memdb v0.0.0-20190306140544-eea0b16292ad h1:Z8ldPkr3NLLacRTNmpAug4Uz09349gcnOB0b7PDD7/0=
github.com/hashicorp/go-memdb v0.0.0-20190306140544-eea0b16292ad/go.mod h1:kbfItVoBJwCfKXDXN4YoAXjxcFVZ7MRrJzyTX6H4giE=
github.com/hashicorp/go-msgpack v0.5.3 h1:zKjpN5BK/P5lMYrLmBHdBULWbJ0XpYR+7NGzqkZzoD4=
github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM=
github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o=
github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk=
github.com/hashicorp/go-plugin v0.0.0-20190220160451-3f118e8ee104 h1:9iQ/zrTOJqzP+kH37s6xNb6T1RysiT7fnDD3DJbspVw=
github.com/hashicorp/go-plugin v0.0.0-20190220160451-3f118e8ee104/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
github.com/hashicorp/go-retryablehttp v0.0.0-20180718195005-e651d75abec6 h1:qCv4319q2q7XKn0MQbi8p37hsJ+9Xo8e6yojA73JVxk=
github.com/hashicorp/go-retryablehttp v0.0.0-20180718195005-e651d75abec6/go.mod h1:fXcdFsQoipQa7mwORhKad5jmDCeSy/RCGzWA08PO0lM=
github.com/hashicorp/go-rootcerts v0.0.0-20160503143440-6bb64b370b90 h1:VBj0QYQ0u2MCJzBfeYXGexnAl17GsH1yidnoxCqqD9E=
github.com/hashicorp/go-rootcerts v0.0.0-20160503143440-6bb64b370b90/go.mod h1:o4zcYY1e0GEZI6eSEr+43QDYmuGglw1qSO6qdHUHCgg=
github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU=
github.com/hashicorp/go-sockaddr v1.0.2 h1:ztczhD1jLxIRjVejw8gFomI1BQZOe2WoVOu0SyteCQc=
github.com/hashicorp/go-sockaddr v1.0.2/go.mod h1:rB4wwRAUzs07qva3c5SdrY/NEtAUjGlgmH/UkBUC97A=
github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4=
github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.1 h1:fv1ep09latC32wFoVwnqcnKJGnMSdBanPczbHAYm1BE=
github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-version v0.0.0-20160214002439-2e7f5ea8e27b h1:j5hhoMJXMZPck42/fNTe6EWsKmkgr8bZyCIGZKUoW2o=
github.com/hashicorp/go-version v0.0.0-20160214002439-2e7f5ea8e27b/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ=
github.com/hashicorp/memberlist v0.1.3 h1:EmmoJme1matNzb+hMpDuR/0sbJSUisxyqBGG676r31M=
github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I=
github.com/hashicorp/nomad v0.8.7 h1:jOrmJdAoWcyhKgoG4OxHQhG5SU6RniXFjfwKg6a492U=
github.com/hashicorp/nomad v0.8.7/go.mod h1:WRaKjdO1G2iqi86TvTjIYtKTyxg4pl7NLr9InxtWaI0=
github.com/hashicorp/raft v1.0.0 h1:htBVktAOtGs4Le5Z7K8SF5H2+oWsQFYVmOgH5loro7Y=
github.com/hashicorp/raft v1.0.0/go.mod h1:DVSAWItjLjTOkVbSpWQ0j0kUADIvDaCtBxIcbNAQLkI=
github.com/hashicorp/serf v0.8.2 h1:YZ7UKsJv+hKjqGVUUbtE3HNj79Eln2oQ75tniF6iPt0=
github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc=
github.com/hashicorp/vault v0.0.0-20181121181053-d4367e581fe1 h1:Z9HQ6wYfPZvwDMPFBMxnAO7SI80iGMq+Hhd2+dOsp84=
github.com/hashicorp/vault v0.0.0-20181121181053-d4367e581fe1/go.mod h1:KfSyffbKxoVyspOdlaGVjIuwLobi07qD1bAbosPMpP0=
github.com/hashicorp/vault-plugin-auth-alicloud v0.0.0-20190311155555-98628998247d h1:c6V5Sn72jhmkJlEsfGUhfcWtRYml/iIxx5FI3Q2NvhE=
github.com/hashicorp/vault-plugin-auth-alicloud v0.0.0-20190311155555-98628998247d/go.mod h1:o3i5QQWgV5+SYouIn++L9D0kbhLYB3FjxNRHNf6KS+Q=
github.com/hashicorp/vault-plugin-auth-azure v0.0.0-20190201222632-0af1d040b5b3 h1:A8VkhfjR3B7PkwQCwIj+0S76xBQMpetKPq12Ve7y6KQ=
github.com/hashicorp/vault-plugin-auth-azure v0.0.0-20190201222632-0af1d040b5b3/go.mod h1:f+VmjSQIxxO+YTeO3FbPWRPCPbd3f3lwpP6jaO/YduQ=
github.com/hashicorp/vault-plugin-auth-centrify v0.0.0-20180816201131-66b0a34a58bf h1:AizyQbOM+h/ZjV0ao65aJ/kQpOxlOKgZORLP/XnJlTA=
github.com/hashicorp/vault-plugin-auth-centrify v0.0.0-20180816201131-66b0a34a58bf/go.mod h1:IIz+CMBKBEFyjeBeFUlpoUuMOyFb7mybOUNP6GX1xuk=
github.com/hashicorp/vault-plugin-auth-gcp v0.0.0-20190201215414-7d4c2101e7d0 h1:xP4Iiq/X4is8ZCVM/A5rrXjHMmEzqZ9+0Smj0Ft8KpQ=
github.com/hashicorp/vault-plugin-auth-gcp v0.0.0-20190201215414-7d4c2101e7d0/go.mod h1:E/E+5CuQCjOn/YGCmZ/tA7GwLey/lN1PwwJOOa9Iqy0=
github.com/hashicorp/vault-plugin-auth-jwt v0.0.0-20190314211503-86b44673ce1e h1:cezAuunlgcIDrKL9L0I72CTL2jctqAIkUzo4oqHJb+0=
github.com/hashicorp/vault-plugin-auth-jwt v0.0.0-20190314211503-86b44673ce1e/go.mod h1:j6Xmkj3dzuC63mivquwVVTlxjwDndwNxi4cJUku40J8=
github.com/hashicorp/vault-plugin-auth-kubernetes v0.0.0-20190201222209-db96aa4ab438 h1:ylEjxR+l2UwEPNiSNYnrpbnqYkmyhGYs3gutLLopbgc=
github.com/hashicorp/vault-plugin-auth-kubernetes v0.0.0-20190201222209-db96aa4ab438/go.mod h1:PqRUU5TaQ6FwVTsHPLrJs1F+T5IjbSzlfTy9cTyGeHM=
github.com/hashicorp/vault-plugin-secrets-ad v0.0.0-20190131222416-4796d9980125 h1:EWUv105dSegjxZacnQ4+DKXHaVnECielqHtw5eOhGNs=
github.com/hashicorp/vault-plugin-secrets-ad v0.0.0-20190131222416-4796d9980125/go.mod h1:4vRQzvp3JI+g4oUqzcklIEj2UKyhQnpIo+BDbh2uzYM=
github.com/hashicorp/vault-plugin-secrets-alicloud v0.0.0-20190131211812-b0abe36195cb h1:mWY3OMYmQqxlbGiu/ffG7UGua3eWUqe2LjyiajyNYLE=
github.com/hashicorp/vault-plugin-secrets-alicloud v0.0.0-20190131211812-b0abe36195cb/go.mod h1:rl8WzY7++fZMLXd6Z/k/o9wUmMbOqpTLhbtKs1loMU0=
github.com/hashicorp/vault-plugin-secrets-azure v0.0.0-20181207232500-0087bdef705a h1:1sAzFWTeuHSBT+IzQmmWCw2oN9ZBNyPGbG2y30TYjK8=
github.com/hashicorp/vault-plugin-secrets-azure v0.0.0-20181207232500-0087bdef705a/go.mod h1:/DhLpYuRP2o00gkj6S0Gy7NvKk5AaAtP6p3f+OmxDUI=
github.com/hashicorp/vault-plugin-secrets-gcp v0.0.0-20190311200649-621231cb86fe h1:MheAGVIKlMmR4m5rslSCeylNszrOqY/bRhWXI3oUdgU=
github.com/hashicorp/vault-plugin-secrets-gcp v0.0.0-20190311200649-621231cb86fe/go.mod h1:IV2OZZZ9FCtSYeKDLsnO5JipMdjwachVISz9pNuQjhs=
github.com/hashicorp/vault-plugin-secrets-gcpkms v0.0.0-20190116164938-d6b25b0b4a39 h1:Tg7YyHyK9P/Z/2aA3NTngjLhZN/1CoeWtbUutoX5A6E=
github.com/hashicorp/vault-plugin-secrets-gcpkms v0.0.0-20190116164938-d6b25b0b4a39/go.mod h1:2n62quNV4DvfMY5Lxx82NJmx+9pYtv4RltLIFKxEO4E=
github.com/hashicorp/vault-plugin-secrets-kv v0.0.0-20190315192709-dccffee64925 h1:QpryTbZYYtc5zE5qZghxzUsBLmT4YtW3LBZMWRmFxXY=
github.com/hashicorp/vault-plugin-secrets-kv v0.0.0-20190315192709-dccffee64925/go.mod h1:VJHHT2SC1tAPrfENQeBhLlb5FbZoKZM+oC/ROmEftz0=
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb h1:b5rjCoWHc7eqmAS4/qyk21ZsHyb6Mxv/jykxvNTkU4M=
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM=
github.com/howeyc/gopass v0.0.0-20170109162249-bf9dde6d0d2c h1:kQWxfPIHVLbgLzphqk3QUflDy9QdksZR4ygR807bpy0=
github.com/howeyc/gopass v0.0.0-20170109162249-bf9dde6d0d2c/go.mod h1:lADxMC39cJJqL93Duh1xhAs4I2Zs8mKS89XWXFGp9cs=
github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/inconshreveable/go-update v0.0.0-20160112193335-8152e7eb6ccf h1:WfD7VjIE6z8dIvMsI4/s+1qr5EL+zoIGev1BQj1eoJ8=
github.com/inconshreveable/go-update v0.0.0-20160112193335-8152e7eb6ccf/go.mod h1:hyb9oH7vZsitZCiBt0ZvifOrB+qc8PS5IiilCIb87rg=
github.com/jeffchao/backoff v0.0.0-20140404060208-9d7fd7aa17f2 h1:mex1izRBCD+7WjieGgRdy7e651vD/lvB1bD9vNE/3K4=
github.com/jeffchao/backoff v0.0.0-20140404060208-9d7fd7aa17f2/go.mod h1:xkfESuHriIekR+4RoV+fu91j/CfnYM29Zi2tMFw5iD4=
github.com/jefferai/jsonx v1.0.0 h1:Xoz0ZbmkpBvED5W9W1B5B/zc3Oiq7oXqiW7iRV3B6EI=
github.com/jefferai/jsonx v1.0.0/go.mod h1:OGmqmi2tTeI/PS+qQfBDToLHHJIy/RMp24fPo8vFvoQ=
github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8 h1:12VvqtR6Aowv3l/EQUlocDHW2Cp4G9WJVH7uyH8QFJE=
github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jonboulle/clockwork v0.1.0 h1:VKV+ZcuP6l3yW9doeqz6ziZGgcynBVQO+obU0+0hcPo=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/json-iterator/go v1.1.6 h1:MrUvLMLTMxbqFJ9kzlvat/rYZqZnW3u4wkLzWTaFwKs=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/jtolds/gls v4.2.1+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/keybase/go-crypto v0.0.0-20190312101036-b475f2ecc1fe h1:1uflwka8xAWVB2vfZgn+EzemfgGi7WA/XHDnECEeSCM=
github.com/keybase/go-crypto v0.0.0-20190312101036-b475f2ecc1fe/go.mod h1:ghbZscTyKdM07+Fw3KSi0hcJm+AlEUWj8QLlPtijN/M=
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.3.0 h1:kKeUSEWOEaY7o6tEo81HGkvA90Q+qCd1WIkOjr7HgvQ=
github.com/klauspost/compress v1.3.0/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/cpuid v0.0.0-20160106104451-349c67577817 h1:/7pPahIC+GoCm/euDCi2Pm29bAj9tc6TcK4Zcc8D3WI=
github.com/klauspost/cpuid v0.0.0-20160106104451-349c67577817/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/klauspost/crc32 v0.0.0-20161016154125-cb6bfca970f6 h1:KAZ1BW2TCmT6PRihDPpocIy1QTtsAsrx6TneU/4+CMg=
github.com/klauspost/crc32 v0.0.0-20161016154125-cb6bfca970f6/go.mod h1:+ZoRqAPRLkC4NPOvfYeR5KNOrY6TD+/sAC3HXPZgDYg=
github.com/klauspost/pgzip v0.0.0-20180606150939-90b2c57fba35 h1:V1xjy9E/9RjLJs+cSLIY0LxUtqc2q1Ufq5O7MhOqfz8=
github.com/klauspost/pgzip v0.0.0-20180606150939-90b2c57fba35/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/klauspost/reedsolomon v0.0.0-20190210214925-2b210cf0866d h1:alKbTsh5QbtSTSX09Xoi13lfftojLoznxBowyexhpkQ=
github.com/klauspost/reedsolomon v0.0.0-20190210214925-2b210cf0866d/go.mod h1:CwCi+NUr9pqSVktrkN+Ondf06rkhYZ/pcNv7fu+8Un4=
github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/lib/pq v0.0.0-20181016162627-9eb73efc1fcc h1:0pifi8wVV/YuUKBDmlH3koJgRVnUJ2RiJQ8ly/1/aJ8=
github.com/lib/pq v0.0.0-20181016162627-9eb73efc1fcc/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/marstr/guid v1.1.0 h1:/M4H/1G4avsieL6BbUwCOBzulmoeKVP5ux/3mQNnbyI=
github.com/marstr/guid v1.1.0/go.mod h1:74gB1z2wpxxInTG6yaqA7KrtM0NZ+RbrcqDvYHefzho=
github.com/mattbaird/elastigo v0.0.0-20170123220020-2fe47fd29e4b h1:v29yPGHhOqw7VHEnTeQFAth3SsBrmwc8JfuhNY0G34k=
github.com/mattbaird/elastigo v0.0.0-20170123220020-2fe47fd29e4b/go.mod h1:5MWrJXKRQyhQdUCF+vu6U5c4nQpg70vW3eHaU0/AYbU=
github.com/mattn/go-colorable v0.0.0-20160210001857-9fdad7c47650/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-colorable v0.0.9 h1:UVL0vNpWh04HeJXV0KLcaT7r06gOH2l4OW6ddYRUIY4=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-isatty v0.0.3 h1:ns/ykhmWi7G9O+8a448SecJU3nSMBXJfqQkl0upE1jI=
github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-isatty v0.0.4 h1:bnP0vzxcAdeI1zdubAl5PjU6zsERjGZb7raWodagDYs=
github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-runewidth v0.0.4 h1:2BvfKmzob6Bmd4YsL0zygOqfdFnK7GR4QL06Do4/p7Y=
github.com/mattn/go-runewidth v0.0.4/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/michaelklishin/rabbit-hole v1.5.0 h1:Bex27BiFDsijCM9D0ezSHqyy0kehpYHuNKaPqq/a4RM=
github.com/michaelklishin/rabbit-hole v1.5.0/go.mod h1:vvI1uOitYZi0O5HEGXhaWC1XT80Gy+HvFheJ+5Krlhk=
github.com/miekg/dns v1.0.14 h1:9jZdLNd/P4+SfEJ0TNyxYpsK8N4GtfylBLqtbYN1sbA=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
github.com/minio/blazer v0.0.0-20171126203752-2081f5bf0465 h1:rV63TOug3ph7F/cpkDSXdZNxyH3LCta+Lk5A/IYkt/0=
github.com/minio/blazer v0.0.0-20171126203752-2081f5bf0465/go.mod h1:ChYiRE5crAmaOo0dQth8FJCtLTiPH9QEMVlDcMp+938=
github.com/minio/cli v0.0.0-20170227073228-b8ae5507c0ce h1:cMutC6C4ElU0TVblNfXx6us/E5J54zgxMCoELhQ+Dj4=
github.com/minio/cli v0.0.0-20170227073228-b8ae5507c0ce/go.mod h1:hLsWNQy2wIf3FKFnMlH69f4RdEyn8nbRA2shaulTjGY=
github.com/minio/dsync v0.0.0-20190104003057-61c41ffdeea2 h1:d244DGqthdKVVB7fKUN/oVK0w1GEP3Da5YUAJKZivU4=
github.com/minio/dsync v0.0.0-20190104003057-61c41ffdeea2/go.mod h1:eLQe3mXL0h02kNpPtBJiLr1fIEIJftgXRAjncjQbxJo=
github.com/minio/highwayhash v0.0.0-20181220011308-93ed73d64169 h1:ysJt+W4oIB0YVMckoJt6SjJN/eF57XQdAlWZw786smU=
github.com/minio/highwayhash v0.0.0-20181220011308-93ed73d64169/go.mod h1:NL8wme5P5MoscwAkXfGroz3VgpCdhBw3KYOu5mEsvpU=
github.com/minio/lsync v0.0.0-20190207022115-a4e43e3d0887 h1:MIpCDz3d2FR2a+FjdizuFdjsoeHuLlSkl3YNQJ55jV8=
github.com/minio/lsync v0.0.0-20190207022115-a4e43e3d0887/go.mod h1:ni10+iSX7FO8N2rv41XM444V6w4rYO0dZo5KIkbn/YA=
github.com/minio/mc v0.0.0-20190311071728-2e612b23d665 h1:fL4R87wJA+Y0RjGMD8gUnI0AkBc5q1+APDDVx3Loz6g=
github.com/minio/mc v0.0.0-20190311071728-2e612b23d665/go.mod h1:7qLZXNjCD55DJ3iqe1uWoUh1MASRVd1M6wnqSdyhx7Y=
github.com/minio/minio v0.0.0-20190206103305-fd4e15c11641/go.mod h1:lXcp05uxYaW99ebgI6ZKIGYU7tqZkM5xSsG0xRt4VIU=
github.com/minio/minio-go v0.0.0-20190227180923-59af836a7e6d h1:gptD0/Hnam7h4Iq9D/33fscRpHfzOOUqUbH2nPw9HcU=
github.com/minio/minio-go v0.0.0-20190227180923-59af836a7e6d/go.mod h1:/haSOWG8hQNx2+JOfLJ9GKp61EAmgPwRVw/Sac0NzaM=
github.com/minio/parquet-go v0.0.0-20190304214338-8747cd6359d3 h1:jICxM8jOUfk4GRAgy5x1KAkTBdaQuZSc5H67nB5BMEw=
github.com/minio/parquet-go v0.0.0-20190304214338-8747cd6359d3/go.mod h1:J+goXSuzlte5imWMqb6cUWC/tbYYysUHctwmKXomYzM=
github.com/minio/parquet-go v0.0.0-20190318185229-9d767baf1679 h1:OMKaN/82sBHUZPvjYNBFituHExa1OGY63eACDGtetKs=
github.com/minio/parquet-go v0.0.0-20190318185229-9d767baf1679/go.mod h1:J+goXSuzlte5imWMqb6cUWC/tbYYysUHctwmKXomYzM=
github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16 h1:5W7KhL8HVF3XCFOweFD3BNESdnO8ewyYTFT2R+/b8FQ=
github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
github.com/minio/sio v0.0.0-20180327104954-6a41828a60f0 h1:ys4bbOlPvaUBlA0byjm6TqydsXZu614ZIUTfF+4MRY0=
github.com/minio/sio v0.0.0-20180327104954-6a41828a60f0/go.mod h1:PDJGYr8GXjiOTIst0hQMOSK5FdXLwObr2cGbiMddDPc=
github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
github.com/mitchellh/copystructure v1.0.0 h1:Laisrj+bAB6b/yJwB5Bt3ITZhGJdqmxquMKeZ+mmkFQ=
github.com/mitchellh/copystructure v1.0.0/go.mod h1:SNtv71yrdKgLRyLFxmLdkAbkKEFWgYaq1OVrnRcwhnw=
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/mitchellh/go-testing-interface v0.0.0-20171004221916-a61a99592b77/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
github.com/mitchellh/go-testing-interface v1.0.0 h1:fzU/JVNcaqHQEcVFAKeR41fkiLdIPrefOvVG1VZ96U0=
github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo=
github.com/mitchellh/gox v0.4.0/go.mod h1:Sd9lOJ0+aimLBi73mGofS1ycjY8lL3uZM3JPS42BGNg=
github.com/mitchellh/hashstructure v1.0.0 h1:ZkRJX1CyOoTkar7p/mLS5TZU4nJ1Rn/F8u9dGS02Q3Y=
github.com/mitchellh/hashstructure v1.0.0/go.mod h1:QjSHrPWS+BGUVBYkbTZWEnOh3G1DutKwClXU/ABz6AQ=
github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0QubkSMEySY=
github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee h1:kK7VuFVykgt0LfMSloWYjDOt4TnOcL0AxF0/rDq2VkM=
github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/mitchellh/pointerstructure v0.0.0-20170205204203-f2329fcfa9e2 h1:KkcG7Ppt1wn2lmtjNr55d1tG4kbDA+TDaWtuezT5WS4=
github.com/mitchellh/pointerstructure v0.0.0-20170205204203-f2329fcfa9e2/go.mod h1:KMNPMpc0BU/kZEgyDhBplsDn/mjnJMhyMjq4MWboN20=
github.com/mitchellh/reflectwalk v1.0.0 h1:9D+8oIskB4VJBN5SFlmc27fSlIBZaov1Wpk/IfikLNY=
github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/nats-io/gnatsd v1.4.1 h1:RconcfDeWpKCD6QIIwiVFcvForlXpWeJP7i5/lDLy44=
github.com/nats-io/gnatsd v1.4.1/go.mod h1:nqco77VO78hLCJpIcVfygDP2rPGfsEHkGTUk94uh5DQ=
github.com/nats-io/go-nats v0.0.0-20161120202126-6b6bf392d34d h1:WB/xFGGdAliPnwnKOUIXp9qLCYQtbh09j2nDgtzI948=
github.com/nats-io/go-nats v0.0.0-20161120202126-6b6bf392d34d/go.mod h1:+t7RHT5ApZebkrQdnn6AhQJmhJJiKAvJUio1PiiCtj0=
github.com/nats-io/go-nats-streaming v0.0.0-20161216191029-077898146bfb h1:jUIMu/j8MvC05arCap7hVlhNStawzBrQaZ2CU+IHrSE=
github.com/nats-io/go-nats-streaming v0.0.0-20161216191029-077898146bfb/go.mod h1:gfq4R3c9sKAINOpelo0gn/b9QDMBZnmrttcsNF+lqyo=
github.com/nats-io/nats v0.0.0-20160916181735-70b70be17b77 h1:3Uk8Cn5GxarM118Ya1pkd7p8Y0DQgNlbhaJaqBOvDwY=
github.com/nats-io/nats v0.0.0-20160916181735-70b70be17b77/go.mod h1:PpmYZwlgTfBI56QypJLfIMOfLnMRuVs+VL6r8mQ2SoQ=
github.com/nats-io/nats-streaming-server v0.12.2 h1:EpyLfUBZgwu5c0mdSSytQsapm615AyitPssq7jgafdw=
github.com/nats-io/nats-streaming-server v0.12.2/go.mod h1:RyqtDJZvMZO66YmyjIYdIvS69zu/wDAkyNWa8PIUa5c=
github.com/nats-io/nuid v1.0.0 h1:44QGdhbiANq8ZCbUkdn6W5bqtg+mHuDE4wOUuxxndFs=
github.com/nats-io/nuid v1.0.0/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
github.com/nsqio/go-nsq v0.0.0-20181028195256-0527e80f3ba5 h1:pQydq98llDs8dmbPtEtgFT8gsKwGnRs811DDKHqWWLA=
github.com/nsqio/go-nsq v0.0.0-20181028195256-0527e80f3ba5/go.mod h1:XP5zaUs3pqf+Q71EqUJs3HYfBIqfK6G83WQMdNN+Ito=
github.com/oklog/run v1.0.0 h1:Ru7dDtJNOyC66gQ5dQmaCa0qIsAUFY3sFpK1Xk8igrw=
github.com/oklog/run v1.0.0/go.mod h1:dlhp/R75TPv97u0XWUtDeV/lRKWPKSdTuV0TZvrmrQA=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/opencontainers/go-digest v1.0.0-rc1 h1:WzifXhOVOEOuFYOJAW6aQqW0TooG2iki3E3Ii+WN7gQ=
github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
github.com/opencontainers/image-spec v1.0.1 h1:JMemWkRwHx4Zj+fVxWoMCFm/8sYGGrUVojFA6h/TRcI=
github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/opencontainers/runc v0.1.1 h1:GlxAyO6x8rfZYN9Tt0Kti5a/cP41iuiO2yYT0IJGY8Y=
github.com/opencontainers/runc v0.1.1/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg=
github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8=
github.com/ory-am/common v0.4.0 h1:edGPoxYX4hno0IJHXh9TCMUPR6ZcJp+y6aClFYxeuUE=
github.com/ory-am/common v0.4.0/go.mod h1:oCYGuwwM8FyYMKqh9vrhBaeUoyz/edx0bgJN6uS6/+k=
github.com/ory/dockertest v3.3.4+incompatible h1:VrpM6Gqg7CrPm3bL4Wm1skO+zFWLbh7/Xb5kGEbJRh8=
github.com/ory/dockertest v3.3.4+incompatible/go.mod h1:1vX4m9wsvi00u5bseYwXaSnhNrne+V0E6LAcBILJdPs=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c h1:Lgl0gzECD8GnQ5QCWA8o6BtfL6mDH5rQgM4/fX3avOs=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pborman/uuid v1.2.0 h1:J7Q5mO4ysT1dv8hyrUGHb9+ooztCXu1D8MY8DZYsu3g=
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
github.com/pierrec/lz4 v2.0.5+incompatible h1:2xWsjqPFWcplujydGg4WmhC/6fZqK42wMM8aXeqhl0I=
github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/profile v0.0.0-20160315012113-19396ecfda37/go.mod h1:hJw3o1OdXxsrSjjVksARp5W95eeEaEfptyVZyv6JUPA=
github.com/pkg/profile v1.2.1 h1:F++O52m40owAmADcojzM+9gyjmMOY/T4oYJkgFDH8RE=
github.com/pkg/profile v1.2.1/go.mod h1:hJw3o1OdXxsrSjjVksARp5W95eeEaEfptyVZyv6JUPA=
github.com/pkg/xattr v0.0.0-20170808190211-56ed87199eba h1:WIqO7Q7EUftmER0kakrexuyEWGq+W8ToUfPM/9oZG28=
github.com/pkg/xattr v0.0.0-20170808190211-56ed87199eba/go.mod h1:wuo6utqb0b/WNJYm0fQyg57cKpORNfpX2lY6Ew6+Grg=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
github.com/posener/complete v1.2.1 h1:LrvDIY//XNo65Lq84G/akBuMGlawHvGBABv8f/ZN6DI=
github.com/posener/complete v1.2.1/go.mod h1:6gapUrK/U1TAN7ciCoNRIdVC5sbdBTUh1DKN0g6uH7E=
github.com/pquerna/cachecontrol v0.0.0-20180517163645-1555304b9b35 h1:J9b7z+QKAmPf4YLrFg6oQUotqHQeUNWwkvo7jZp1GLU=
github.com/pquerna/cachecontrol v0.0.0-20180517163645-1555304b9b35/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA=
github.com/pquerna/otp v1.1.0 h1:q2gMsMuMl3JzneUaAX1MRGxLvOG6bzXV51hivBaStf0=
github.com/pquerna/otp v1.1.0/go.mod h1:Zad1CMQfSQZI5KLpahDiSUX4tMMREnXw98IvL1nhgMk=
github.com/prometheus/client_golang v0.8.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.2 h1:awm861/B8OKDd2I/6o1dy3ra4BamzKhYOiGItCeZ740=
github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 h1:idejC8f05m9MGOsuEi1ATq9shN03HrxNkD/luQvxCv8=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 h1:S/YWwWx/RA8rT8tKFRuGUZhuA90OyIBpPCXkcbwU8DE=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e h1:n/3MEhJQjQxrOUCzh1Y3Re6aJUUWRp2M9+Oc3eVn/54=
github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.0.0-20181126121408-4724e9255275 h1:PnBWHBf+6L0jOqq0gIVUe6Yk0/QMZ640k6NvkxcBf+8=
github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273 h1:agujYaXJSxSo18YNX3jzl+4G6Bstwt+kqv47GS12uL0=
github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a h1:9a8MnZMP0X2nLJdBg+pBmGgkJlSaKC2KaQmTCk1XDtE=
github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/rjeczalik/notify v0.9.2 h1:MiTWrPj55mNDHEiIX5YUSKefw/+lCQVoAFmD6oQm5w8=
github.com/rjeczalik/notify v0.9.2/go.mod h1:aErll2f0sUX9PXZnVNyeiObbmTlk5jnMoCa4QEjJeqM=
github.com/rs/cors v0.0.0-20190116175910-76f58f330d76 h1:kz+slcZ3xepXoLw24pyf3+fnc3WJITZ91IEa+PJTv2g=
github.com/rs/cors v0.0.0-20190116175910-76f58f330d76/go.mod h1:gFx+x8UowdsKA9AchylcLynDq+nNFfI8FkUZdN/jGCU=
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
github.com/ryanuber/columnize v2.1.0+incompatible/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
github.com/ryanuber/go-glob v1.0.0 h1:iQh3xXAumdQ+4Ufa5b25cRpC5TYKlno6hsv6Cb3pkBk=
github.com/ryanuber/go-glob v1.0.0/go.mod h1:807d1WSdnB0XRJzKNil9Om6lcp/3a0v4qIHxIXzX/Yc=
github.com/samuel/go-zookeeper v0.0.0-20180130194729-c4fab1ac1bec h1:6ncX5ko6B9LntYM0YBRXkiSaZMmLYeZ/NWcmeB43mMY=
github.com/samuel/go-zookeeper v0.0.0-20180130194729-c4fab1ac1bec/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E=
github.com/satori/go.uuid v1.2.0 h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 h1:nn5Wsu0esKSJiIVhscUtVbo7ada43DJhG55ua/hjS5I=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/segmentio/go-prompt v0.0.0-20161017233205-f0d19b6901ad h1:2ZO86bDscwpNrDfu3R0YlSF1igDzsk0EdrLzwgJqqUQ=
github.com/segmentio/go-prompt v0.0.0-20161017233205-f0d19b6901ad/go.mod h1:B3ehdD1xPoWDKgrQgUaGk+m8H1xb1J5TyYDfKpKNeEE=
github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
github.com/sirupsen/logrus v1.3.0 h1:hI/7Q+DtNZ2kINb6qt/lS+IyXnHQe9e90POfeewL/ME=
github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/skyrings/skyring-common v0.0.0-20160324141443-762fd2bfc12e h1:oZ1fpzB0OtG3e9d1NbnERBTvQGx5NDZ1pr2kWPZbGUI=
github.com/skyrings/skyring-common v0.0.0-20160324141443-762fd2bfc12e/go.mod h1:d8hQseuYt4rJoOo21lFzYJdhMjmDqLY++ayArbgYjWI=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/assertions v0.0.0-20190116191733-b6c0e53d7304/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/assertions v0.0.0-20190215210624-980c5ac6f3ac h1:wbW+Bybf9pXxnCFAOWZTqkRjAc7rAIwo2e1ArUhiHxg=
github.com/smartystreets/assertions v0.0.0-20190215210624-980c5ac6f3ac/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s=
github.com/smartystreets/goconvey v0.0.0-20190222223459-a17d461953aa/go.mod h1:2RVY1rIf+2J2o/IM9+vPq9RzmHDSseB7FoXiSNIUsoU=
github.com/smartystreets/goconvey v0.0.0-20190306220146-200a235640ff h1:86HlEv0yBCry9syNuylzqznKXDK11p6D0DT596yNMys=
github.com/smartystreets/goconvey v0.0.0-20190306220146-200a235640ff/go.mod h1:KSQcGKpxUMHk3nbYzs/tIBAM2iDooCn0BmttHOJEbLs=
github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E=
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/streadway/amqp v0.0.0-20160311215503-2e25825abdbd h1:625/bJvSNfQrzzK5ttnUqMqnVe8M5MILmf5ZRGgeeDY=
github.com/streadway/amqp v0.0.0-20160311215503-2e25825abdbd/go.mod h1:1WNBiOZtZQLpVAyu0iTduoJL9hEsMloAK5XWrtW0xdY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.1 h1:2vfRuCMp5sSVIDSqO8oNnWJq7mPa6KVP3iPIwFBuy8A=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/tidwall/gjson v1.1.2/go.mod h1:c/nTNbUr0E0OrXEhq1pwa8iEgc2DOt4ZZqAt1HtCkPA=
github.com/tidwall/gjson v1.1.4 h1:lonRDhK9sFzw7ogkERBgx5wF6lRP2bpjr6jiwVzYjYc=
github.com/tidwall/gjson v1.1.4/go.mod h1:c/nTNbUr0E0OrXEhq1pwa8iEgc2DOt4ZZqAt1HtCkPA=
github.com/tidwall/match v1.0.0 h1:Ym1EcFkp+UQ4ptxfWlW+iMdq5cPH5nEuGzdf/Pb7VmI=
github.com/tidwall/match v1.0.0/go.mod h1:LujAq0jyVjBy028G1WhWfIzbpQfMO8bBZ6Tyb0+pL9E=
github.com/tidwall/sjson v1.0.4 h1:UcdIRXff12Lpnu3OLtZvnc03g4vH2suXDXhBwBqmzYg=
github.com/tidwall/sjson v1.0.4/go.mod h1:bURseu1nuBkFpIES5cz6zBtjmYeOQmEESshn7VpF15Y=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 h1:LnC5Kc/wtumK+WB441p7ynQJzVuNRJiqddSIE3IlSEQ=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/ugorji/go v0.0.0-20180628102755-7d51bbe6161d h1:vqHumCiddnEi8M6ztdQ9QZkiAJn5fASghAY2A7Jeyic=
github.com/ugorji/go v0.0.0-20180628102755-7d51bbe6161d/go.mod h1:hnLbHMwcvSihnDhEfx2/BzKp2xb0Y+ErdfYcrs9tkJQ=
github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a h1:0R4NLDRDZX6JcmhJgXi5E4b8Wg84ihbmUKp/GvSPEzc=
github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV9WzVtRmSR+PDvWpU/qWl4Wa5LApYYX4ZtKbio=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
go.etcd.io/bbolt v1.3.2 h1:Z/90sZLPOeCy2PwprqkFa25PdkusRzaj9P8zm/KNyvk=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.opencensus.io v0.15.0 h1:r1SzcjSm4ybA0qZs3B4QYX072f8gK61Kh0qtwyFpfdk=
go.opencensus.io v0.15.0/go.mod h1:UffZAU+4sDEINUGP/B7UfBBkq4fqLu9zXAX7ke6CHW0=
go.opencensus.io v0.18.1-0.20181204023538-aab39bd6a98b h1:6ayHMBPtdP3jNuk+Sfhso+PTB7ZJQ5E1FBo403m2H8w=
go.opencensus.io v0.18.1-0.20181204023538-aab39bd6a98b/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
go.uber.org/atomic v1.3.2 h1:2Oa65PReHzfn29GpvgsYwloV9AVFHPDk8tYxt2c2tr4=
go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/zap v1.9.1 h1:XCJQEf3W6eZaVwhRBof6ImoYGJSITeKWsyeh3HFu/5o=
go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
golang.org/x/crypto v0.0.0-20180820150726-614d502a4dac/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190128193316-c7b33c32a30b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190131182504-b8fe1690c613/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a h1:YX8ljsm6wXlHZO+aRz9Exqr0evNhKRNe5K/gi+zKh4U=
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3 h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/net v0.0.0-20180530234432-1e491301e022/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190301231341-16b79f2e4e95/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311175648-12eef18f757f h1:5XQlNmCmC6QCwdP9jITSaPHwflzXW/v2wDg4wSiPscs=
golang.org/x/net v0.0.0-20190311175648-12eef18f757f/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190318221613-d196dffd7c2b h1:ZWpVMTsK0ey5WJCu+vVdfMldWq7/ezaOcjnKWIHWVkE=
golang.org/x/net v0.0.0-20190318221613-d196dffd7c2b/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/oauth2 v0.0.0-20180603041954-1e0a3fa8ba9a/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be h1:vEDujvNQGv4jgYKudGeI/+DAX4Jffq6hpD55MmoEvKs=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f h1:Bl/8QSvNqXvPGPGXa2z5xUTmV7VDcZyvRZ+QQXkXTZQ=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4 h1:YUO/7uOKsKeq9UokNS62b8FYywz3ker1l1vDZRCRefw=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180828065106-d99a578cf41b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180926160741-c2ed4eda69e7/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190124100055-b90733256f2e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190129075346-302c3dd5f1cc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190304154630-e844e0132e93/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190311152110-c8c8c57fd1e1 h1:FQNj2xvjQ1lgFyzbSybGZr792Y8Dy95D7uuqnZAzNaA=
golang.org/x/sys v0.0.0-20190311152110-c8c8c57fd1e1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190318195719-6c81ef8f67ca h1:o2TLx1bGN3W+Ei0EMU5fShLupLmTOU95KvJJmfYhAzM=
golang.org/x/sys v0.0.0-20190318195719-6c81ef8f67ca/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c h1:fqgJT0MGcGpPgpWU7VRdRjuArfcOvC4AoJmILihzhDg=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190318200714-bb1270c20edf h1:OVQ7iQXiQQT4WuYg+7S/bOVVlASHvL1Chsc15Qtkogo=
golang.org/x/tools v0.0.0-20190318200714-bb1270c20edf/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
google.golang.org/api v0.0.0-20180603000442-8e296ef26005/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.0.0-20180916000451-19ff8768a5c0 h1:AJOCn+ScmtWxp6yySbxsiNXi+RrZaHqWgYbZUzl6oLc=
google.golang.org/api v0.0.0-20180916000451-19ff8768a5c0/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/appengine v1.0.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.1.0 h1:igQkv0AAhEIvTEpD5LIpAfav2eeVO9HBTjvKHVJPRSs=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/genproto v0.0.0-20180601223552-81158efcc9f2/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180918203901-c3f76f3b92d1 h1:9JN/x8g8h4eMemKiniLwss/+Avp9VmS+Es8eMU4Ifog=
google.golang.org/genproto v0.0.0-20180918203901-c3f76f3b92d1/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/grpc v1.12.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.14.0 h1:ArxJuB1NWfPY6r9Gp9gqwplT0Ge7nqv9msgu03lHLmo=
google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.15.0/go.mod h1:0JHn/cJsOMiMfNA9+DeHDlAU7KAAB5GDlYFpa9MZMio=
google.golang.org/grpc v1.18.0 h1:IZl7mfBGfbhYx2p2rKRtYgDFw6SBz+kclmxYrCksPPA=
google.golang.org/grpc v1.18.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
gopkg.in/Shopify/sarama.v1 v1.10.1 h1:SoBkWk2qrgFuiAVEGOYlw63FZcS6YLZYqo/z7wFOW2A=
gopkg.in/Shopify/sarama.v1 v1.10.1/go.mod h1:AxnvoaevB2nBjNK17cG61A3LleFcWFwVBHBt+cot4Oc=
gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d h1:TxyelI5cVkbREznMhfzycHdkp5cLA7DpE+GKjSslYhM=
gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d/go.mod h1:cuepJuh7vyXfUyUwEgHQXw849cJrilpS5NeIjOWESAw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/cheggaaa/pb.v1 v1.0.27 h1:kJdccidYzt3CaHD1crCFTS1hxyhSi059NhOFUf03YFo=
gopkg.in/cheggaaa/pb.v1 v1.0.27/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw=
gopkg.in/fatih/pool.v2 v2.0.0 h1:xIFeWtxifuQJGk/IEPKsTduEKcKvPmhoiVDGpC40nKg=
gopkg.in/fatih/pool.v2 v2.0.0/go.mod h1:8xVGeu1/2jr2wm5V9SPuMht2H5AEmf5aFMGSQixtjTY=
gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/gorethink/gorethink.v4 v4.1.0 h1:xoE9qJ9Ae9KdKEsiQGCF44u2JdnjyohrMBRDtts3Gjw=
gopkg.in/gorethink/gorethink.v4 v4.1.0/go.mod h1:M7JgwrUAmshJ3iUbEK0Pt049MPyPK+CYDGGaEjdZb/c=
gopkg.in/h2non/filetype.v1 v1.0.3 h1:EhZ9p3H8eDdFHiKljxJ59EeQ9Pu88wrgY7/B1WRK/VE=
gopkg.in/h2non/filetype.v1 v1.0.3/go.mod h1:M0yem4rwSX5lLVrkEuRRp2/NinFMD5vgJ4DlAhZcfNo=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/ini.v1 v1.41.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/ini.v1 v1.42.0 h1:7N3gPTt50s8GuLortA00n8AqRTk75qOP98+mTPpgzRk=
gopkg.in/ini.v1 v1.42.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce h1:xcEWjVhvbDy+nHP67nPDDpbYrY+ILlfndk4bRioVHaU=
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA=
gopkg.in/olivere/elastic.v5 v5.0.31 h1:MNyEvLWhMubs7rJBkhxwloHJ9VoXMf12G+CPH6e+ZfM=
gopkg.in/olivere/elastic.v5 v5.0.31/go.mod h1:FylZT6jQWtfHsicejzOm3jIMVPOAksa80i3o+6qtQRk=
gopkg.in/ory-am/dockertest.v2 v2.2.3 h1:vSYvP7tvyfAm9merq0gHmcI4yk5nkPpfXmoBCnSP3/4=
gopkg.in/ory-am/dockertest.v2 v2.2.3/go.mod h1:kDHEsan1UcKFYH1c28sDmqnmeqIpB4Nj682gSNhYDYM=
gopkg.in/square/go-jose.v2 v2.3.0 h1:nLzhkFyl5bkblqYBoiWJUt5JkWOzmiaBtCxdJAqJd3U=
gopkg.in/square/go-jose.v2 v2.3.0/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
k8s.io/api v0.0.0-20190313115550-3c12c96769cc h1:m/JS6kQd00rICnXLWlnJzMFQB4AplcURUopS8dKiWmI=
k8s.io/api v0.0.0-20190313115550-3c12c96769cc/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA=
k8s.io/apimachinery v0.0.0-20190313115320-c9defaaddf6f h1:6ojhffWUv9DZ8i4L2LIvSjbWH3fXfP6PmrTNwXHHMhM=
k8s.io/apimachinery v0.0.0-20190313115320-c9defaaddf6f/go.mod h1:ccL7Eh7zubPUSh9A3USN90/OzHNSVN6zxzde07TDCL0=
k8s.io/klog v0.2.0 h1:0ElL0OHzF3N+OhoJTL0uca20SxtYt4X4+bzHeqrB83c=
k8s.io/klog v0.2.0/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk=
layeh.com/radius v0.0.0-20190118135028-0f678f039617 h1:UfoQTGVcI2tUZdQxp4kyh07KW0RB+HrUez9LeHurAvs=
layeh.com/radius v0.0.0-20190118135028-0f678f039617/go.mod h1:fywZKyu//X7iRzaxLgPWsvc0L26IUpVvE/aeIL2JtIQ=
sigs.k8s.io/yaml v1.1.0 h1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs=
sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o=
+24 -18
View File
@@ -35,6 +35,8 @@ import (
// ErrNoEntriesFound - Indicates no entries were found for the given key (directory)
var ErrNoEntriesFound = errors.New("No entries found for this key")
const etcdPathSeparator = "/"
// create a new coredns service record for the bucket.
func newCoreDNSMsg(bucket, ip string, port int, ttl uint32) ([]byte, error) {
return json.Marshal(&SrvRecord{
@@ -73,6 +75,15 @@ func (c *coreDNS) Get(bucket string) ([]SrvRecord, error) {
return srvRecords, nil
}
// msgUnPath converts a etcd path to domainname.
func msgUnPath(s string) string {
ks := strings.Split(strings.Trim(s, etcdPathSeparator), etcdPathSeparator)
for i, j := 0, len(ks)-1; i < j; i, j = i+1, j-1 {
ks[i], ks[j] = ks[j], ks[i]
}
return strings.Join(ks, ".")
}
// Retrieves list of entries under the key passed.
// Note that this method fetches entries upto only two levels deep.
func (c *coreDNS) list(key string) ([]SrvRecord, error) {
@@ -83,7 +94,7 @@ func (c *coreDNS) list(key string) ([]SrvRecord, error) {
return nil, err
}
if r.Count == 0 {
key = strings.TrimSuffix(key, "/")
key = strings.TrimSuffix(key, etcdPathSeparator)
r, err = c.etcdClient.Get(ctx, key)
if err != nil {
return nil, err
@@ -101,23 +112,18 @@ func (c *coreDNS) list(key string) ([]SrvRecord, error) {
}
srvRecord.Key = strings.TrimPrefix(string(n.Key), key)
srvRecord.Key = strings.TrimSuffix(srvRecord.Key, srvRecord.Host)
// SRV records are stored in the following form
// /skydns/net/miniocloud/bucket1, so this function serves multiple
// purposes basically when we do a Get(bucketName) this function
// should return a single DNS record for any input 'bucketName'.
//
// In all other situations when we want to list all DNS records,
// which is handled in the else clause.
for _, domainName := range c.domainNames {
if key != msg.Path(fmt.Sprintf(".%s.", domainName), defaultPrefixPath) {
if srvRecord.Key == "/" {
srvRecords = append(srvRecords, srvRecord)
}
} else {
srvRecords = append(srvRecords, srvRecord)
}
// Skip non-bucket entry like for a key
// /skydns/net/miniocloud/10.0.0.1 that may exist as
// dns entry for the server (rather than the bucket
// itself).
if srvRecord.Key == "" {
continue
}
srvRecord.Key = msgUnPath(srvRecord.Key)
srvRecords = append(srvRecords, srvRecord)
}
if len(srvRecords) == 0 {
return nil, ErrNoEntriesFound
@@ -137,7 +143,7 @@ func (c *coreDNS) Put(bucket string) error {
}
for _, domainName := range c.domainNames {
key := msg.Path(fmt.Sprintf("%s.%s", bucket, domainName), defaultPrefixPath)
key = key + "/" + ip
key = key + etcdPathSeparator + ip
ctx, cancel := context.WithTimeout(context.Background(), defaultContextTimeout)
_, err = c.etcdClient.Put(ctx, key, string(bucketMsg))
defer cancel()
@@ -162,7 +168,7 @@ func (c *coreDNS) Delete(bucket string) error {
}
for _, record := range srvRecords {
dctx, dcancel := context.WithTimeout(context.Background(), defaultContextTimeout)
if _, err = c.etcdClient.Delete(dctx, key+"/"+record.Host); err != nil {
if _, err = c.etcdClient.Delete(dctx, key+etcdPathSeparator+record.Host); err != nil {
dcancel()
return err
}
-78
View File
@@ -1,78 +0,0 @@
// +build ignore
package main
/*
* Minio Cloud Storage, (C) 2017 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
import (
"fmt"
"log"
"github.com/minio/minio/pkg/madmin"
)
func main() {
// Note: YOUR-ACCESSKEYID, YOUR-SECRETACCESSKEY are
// dummy values, please replace them with original values.
// API requests are secure (HTTPS) if secure=true and insecure (HTTPS) otherwise.
// New returns an Minio Admin client object.
madmClnt, err := madmin.New("your-minio.example.com:9000", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", true)
if err != nil {
log.Fatalln(err)
}
bucket := "mybucket"
prefix := "myprefix"
// Create a done channel to control 'ListObjectsHeal' go routine.
doneCh := make(chan struct{})
// Indicate to our routine to exit cleanly upon return.
defer close(doneCh)
// Set true if recursive listing is needed.
isRecursive := true
// List objects that need healing for a given bucket and
// prefix.
healObjectsCh, err := madmClnt.ListObjectsHeal(bucket, prefix, isRecursive, doneCh)
if err != nil {
log.Fatalln(err)
}
for object := range healObjectsCh {
if object.Err != nil {
log.Fatalln(err)
return
}
if object.HealObjectInfo != nil {
switch healInfo := *object.HealObjectInfo; healInfo.Status {
case madmin.CanHeal:
fmt.Println(object.Key, " can be healed.")
case madmin.CanPartiallyHeal:
fmt.Println(object.Key, " can't be healed completely, some disks are offline.")
case madmin.QuorumUnavailable:
fmt.Println(object.Key, " can't be healed until quorum is available.")
case madmin.Corrupted:
fmt.Println(object.Key, " can't be healed, not enough information.")
}
}
fmt.Println("object: ", object)
}
}
+14 -3
View File
@@ -26,11 +26,22 @@ import (
"time"
)
// HealScanMode represents the type of healing scan
type HealScanMode int
const (
// HealNormalScan checks if parts are present and not outdated
HealNormalScan HealScanMode = iota
// HealDeepScan checks for parts bitrot checksums
HealDeepScan
)
// HealOpts - collection of options for a heal sequence
type HealOpts struct {
Recursive bool `json:"recursive"`
DryRun bool `json:"dryRun"`
Remove bool `json:"remove"`
Recursive bool `json:"recursive"`
DryRun bool `json:"dryRun"`
Remove bool `json:"remove"`
ScanMode HealScanMode `json:"scanMode"`
}
// HealStartSuccess - holds information about a successfully started
+6 -5
View File
@@ -72,11 +72,12 @@ type StorageInfo struct {
// ServerProperties holds some of the server's information such as uptime,
// version, region, ..
type ServerProperties struct {
Uptime time.Duration `json:"uptime"`
Version string `json:"version"`
CommitID string `json:"commitID"`
Region string `json:"region"`
SQSARN []string `json:"sqsARN"`
Uptime time.Duration `json:"uptime"`
Version string `json:"version"`
CommitID string `json:"commitID"`
DeploymentID string `json:"deploymentID"`
Region string `json:"region"`
SQSARN []string `json:"sqsARN"`
}
// ServerConnStats holds network information
+16 -13
View File
@@ -19,10 +19,12 @@ package csv
import (
"bytes"
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"github.com/bcicen/jstream"
"github.com/minio/minio/pkg/s3select/sql"
"github.com/tidwall/sjson"
)
// Record - is CSV record.
@@ -78,20 +80,21 @@ func (r *Record) MarshalCSV(fieldDelimiter rune) ([]byte, error) {
// MarshalJSON - encodes to JSON data.
func (r *Record) MarshalJSON() ([]byte, error) {
data := "{}"
var err error
for i := len(r.columnNames) - 1; i >= 0; i-- {
if i >= len(r.csvRecord) {
continue
}
if data, err = sjson.Set(data, r.columnNames[i], r.csvRecord[i]); err != nil {
return nil, err
}
var kvs jstream.KVS = make([]jstream.KV, len(r.columnNames))
for i := 0; i < len(r.columnNames); i++ {
kvs[i] = jstream.KV{Key: r.columnNames[i], Value: r.csvRecord[i]}
}
return json.Marshal(kvs)
}
return []byte(data), nil
// Raw - returns the underlying data with format info.
func (r *Record) Raw() (sql.SelectObjectFormat, interface{}) {
return sql.SelectFmtCSV, r
}
// Replace - is not supported for CSV
func (r *Record) Replace(_ jstream.KVS) error {
return errors.New("Replace is not supported for CSV")
}
// NewRecord - creates new CSV record.
+11 -14
View File
@@ -17,13 +17,11 @@
package json
import (
"encoding/json"
"io"
"github.com/minio/minio/pkg/s3select/sql"
"github.com/bcicen/jstream"
"github.com/tidwall/sjson"
)
// Reader - JSON record reader for S3Select.
@@ -44,23 +42,22 @@ func (r *Reader) Read() (sql.Record, error) {
return nil, io.EOF
}
var data []byte
var err error
var kvs jstream.KVS
if v.ValueType == jstream.Object {
data, err = json.Marshal(v.Value)
// This is a JSON object type (that preserves key
// order)
kvs = v.Value.(jstream.KVS)
} else {
// To be AWS S3 compatible
// Select for JSON needs to output non-object JSON as single column value
// i.e. a map with `_1` as key and value as the non-object.
data, err = sjson.SetBytes(data, "_1", v.Value)
}
if err != nil {
return nil, errJSONParsingError(err)
// To be AWS S3 compatible Select for JSON needs to
// output non-object JSON as single column value
// i.e. a map with `_1` as key and value as the
// non-object.
kvs = jstream.KVS{jstream.KV{Key: "_1", Value: v.Value}}
}
return &Record{
data: data,
KVS: kvs,
SelectFormat: sql.SelectFmtJSON,
}, nil
}
+51 -30
View File
@@ -19,40 +19,40 @@ package json
import (
"bytes"
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"strings"
"github.com/bcicen/jstream"
"github.com/minio/minio/pkg/s3select/sql"
"github.com/tidwall/gjson"
"github.com/tidwall/sjson"
)
// RawJSON is a byte-slice that contains valid JSON
type RawJSON []byte
// MarshalJSON instance for []byte that assumes that byte-slice is
// already serialized JSON
func (b RawJSON) MarshalJSON() ([]byte, error) {
return b, nil
}
// Record - is JSON record.
type Record struct {
data []byte
// Used in Set(), Marshal*()
KVS jstream.KVS
SelectFormat sql.SelectObjectFormat
}
// Get - gets the value for a column name.
func (r *Record) Get(name string) (*sql.Value, error) {
result := gjson.GetBytes(r.data, name)
switch result.Type {
case gjson.Null:
return sql.FromNull(), nil
case gjson.False:
return sql.FromBool(false), nil
case gjson.Number:
return sql.FromFloat(result.Float()), nil
case gjson.String:
return sql.FromString(result.String()), nil
case gjson.True:
return sql.FromBool(true), nil
}
return nil, fmt.Errorf("unsupported gjson value %v; %v", result, result.Type)
// Get is implemented directly in the sql package.
return nil, errors.New("not implemented here")
}
// Set - sets the value for a column name.
func (r *Record) Set(name string, value *sql.Value) (err error) {
func (r *Record) Set(name string, value *sql.Value) error {
var v interface{}
if b, ok := value.ToBool(); ok {
v = b
@@ -67,24 +67,33 @@ func (r *Record) Set(name string, value *sql.Value) (err error) {
} else if value.IsNull() {
v = nil
} else if b, ok := value.ToBytes(); ok {
v = string(b)
v = RawJSON(b)
} else {
return fmt.Errorf("unsupported sql value %v and type %v", value, value.GetTypeString())
}
name = strings.Replace(name, "*", "__ALL__", -1)
r.data, err = sjson.SetBytes(r.data, name, v)
return err
r.KVS = append(r.KVS, jstream.KV{Key: name, Value: v})
return nil
}
// MarshalCSV - encodes to CSV data.
func (r *Record) MarshalCSV(fieldDelimiter rune) ([]byte, error) {
var csvRecord []string
result := gjson.ParseBytes(r.data)
result.ForEach(func(key, value gjson.Result) bool {
csvRecord = append(csvRecord, value.String())
return true
})
for _, kv := range r.KVS {
var columnValue string
switch val := kv.Value.(type) {
case bool, float64, int64, string:
columnValue = fmt.Sprintf("%v", val)
case nil:
columnValue = ""
case RawJSON:
columnValue = string([]byte(val))
default:
return nil, errors.New("Cannot marshal unhandled type")
}
csvRecord = append(csvRecord, columnValue)
}
buf := new(bytes.Buffer)
w := csv.NewWriter(buf)
@@ -101,14 +110,26 @@ func (r *Record) MarshalCSV(fieldDelimiter rune) ([]byte, error) {
return data[:len(data)-1], nil
}
// Raw - returns the underlying representation.
func (r *Record) Raw() (sql.SelectObjectFormat, interface{}) {
return r.SelectFormat, r.KVS
}
// MarshalJSON - encodes to JSON data.
func (r *Record) MarshalJSON() ([]byte, error) {
return r.data, nil
return json.Marshal(r.KVS)
}
// Replace the underlying buffer of json data.
func (r *Record) Replace(k jstream.KVS) error {
r.KVS = k
return nil
}
// NewRecord - creates new empty JSON record.
func NewRecord() *Record {
func NewRecord(f sql.SelectObjectFormat) *Record {
return &Record{
data: []byte("{}"),
KVS: jstream.KVS{},
SelectFormat: f,
}
}
+22 -25
View File
@@ -19,7 +19,8 @@ package parquet
import (
"io"
"github.com/minio/minio/pkg/s3select/json"
"github.com/bcicen/jstream"
jsonfmt "github.com/minio/minio/pkg/s3select/json"
"github.com/minio/minio/pkg/s3select/sql"
parquetgo "github.com/minio/parquet-go"
parquetgen "github.com/minio/parquet-go/gen-go/parquet"
@@ -27,13 +28,13 @@ import (
// Reader - Parquet record reader for S3Select.
type Reader struct {
args *ReaderArgs
file *parquetgo.File
args *ReaderArgs
reader *parquetgo.Reader
}
// Read - reads single record.
func (r *Reader) Read() (rec sql.Record, rerr error) {
parquetRecord, err := r.file.Read()
parquetRecord, err := r.reader.Read()
if err != nil {
if err != io.EOF {
return nil, errParquetParsingError(err)
@@ -42,52 +43,48 @@ func (r *Reader) Read() (rec sql.Record, rerr error) {
return nil, err
}
record := json.NewRecord()
kvs := jstream.KVS{}
f := func(name string, v parquetgo.Value) bool {
if v.Value == nil {
if err := record.Set(name, sql.FromNull()); err != nil {
rerr = errParquetParsingError(err)
}
return rerr == nil
kvs = append(kvs, jstream.KV{Key: name, Value: nil})
return true
}
var value *sql.Value
var value interface{}
switch v.Type {
case parquetgen.Type_BOOLEAN:
value = sql.FromBool(v.Value.(bool))
value = v.Value.(bool)
case parquetgen.Type_INT32:
value = sql.FromInt(int64(v.Value.(int32)))
value = int64(v.Value.(int32))
case parquetgen.Type_INT64:
value = sql.FromInt(int64(v.Value.(int64)))
value = int64(v.Value.(int64))
case parquetgen.Type_FLOAT:
value = sql.FromFloat(float64(v.Value.(float32)))
value = float64(v.Value.(float32))
case parquetgen.Type_DOUBLE:
value = sql.FromFloat(v.Value.(float64))
value = v.Value.(float64)
case parquetgen.Type_INT96, parquetgen.Type_BYTE_ARRAY, parquetgen.Type_FIXED_LEN_BYTE_ARRAY:
value = sql.FromString(string(v.Value.([]byte)))
value = string(v.Value.([]byte))
default:
rerr = errParquetParsingError(nil)
return false
}
if err = record.Set(name, value); err != nil {
rerr = errParquetParsingError(err)
}
return rerr == nil
kvs = append(kvs, jstream.KV{Key: name, Value: value})
return true
}
parquetRecord.Range(f)
return record, rerr
return &jsonfmt.Record{KVS: kvs, SelectFormat: sql.SelectFmtParquet}, nil
}
// Close - closes underlaying readers.
func (r *Reader) Close() error {
return r.file.Close()
return r.reader.Close()
}
// NewReader - creates new Parquet reader using readerFunc callback.
func NewReader(getReaderFunc func(offset, length int64) (io.ReadCloser, error), args *ReaderArgs) (*Reader, error) {
file, err := parquetgo.Open(getReaderFunc, nil)
reader, err := parquetgo.NewReader(getReaderFunc, nil)
if err != nil {
if err != io.EOF {
return nil, errParquetParsingError(err)
@@ -97,7 +94,7 @@ func NewReader(getReaderFunc func(offset, length int64) (io.ReadCloser, error),
}
return &Reader{
args: args,
file: file,
args: args,
reader: reader,
}, nil
}
+5 -1
View File
@@ -246,7 +246,7 @@ func (s3Select *S3Select) outputRecord() sql.Record {
case csvFormat:
return csv.NewRecord()
case jsonFormat:
return json.NewRecord()
return json.NewRecord(sql.SelectFmtJSON)
}
panic(fmt.Errorf("unknown output format '%v'", s3Select.Output.format))
@@ -391,6 +391,10 @@ func (s3Select *S3Select) Evaluate(w http.ResponseWriter) {
break
}
if inputRecord, err = s3Select.statement.EvalFrom(s3Select.Input.format, inputRecord); err != nil {
break
}
if s3Select.statement.IsAggregated() {
if err = s3Select.statement.AggregateRow(inputRecord); err != nil {
break
+12 -1
View File
@@ -16,6 +16,8 @@
package sql
import "fmt"
type s3Error struct {
code string
message string
@@ -91,7 +93,16 @@ func errQueryAnalysisFailure(err error) *s3Error {
func errBadTableName(err error) *s3Error {
return &s3Error{
code: "BadTableName",
message: "The table name is not supported",
message: fmt.Sprintf("The table name is not supported: %v", err),
statusCode: 400,
cause: err,
}
}
func errDataSource(err error) *s3Error {
return &s3Error{
code: "DataSourcePathUnsupported",
message: fmt.Sprintf("Data source: %v", err),
statusCode: 400,
cause: err,
}
+41 -1
View File
@@ -17,8 +17,11 @@
package sql
import (
"encoding/json"
"errors"
"strings"
"github.com/bcicen/jstream"
)
var (
@@ -319,7 +322,44 @@ func (e *JSONPath) evalNode(r Record) (*Value, error) {
if len(ps) == 2 {
keypath = ps[1]
}
return r.Get(keypath)
objFmt, rawVal := r.Raw()
switch objFmt {
case SelectFmtJSON, SelectFmtParquet:
rowVal := rawVal.(jstream.KVS)
pathExpr := e.PathExpr
if len(pathExpr) == 0 {
pathExpr = []*JSONPathElement{{Key: &ObjectKey{ID: e.BaseKey}}}
}
result, err := jsonpathEval(pathExpr, rowVal)
if err != nil {
return nil, err
}
switch rval := result.(type) {
case string:
return FromString(rval), nil
case float64:
return FromFloat(rval), nil
case int64:
return FromInt(rval), nil
case bool:
return FromBool(rval), nil
case jstream.KVS, []interface{}:
bs, err := json.Marshal(result)
if err != nil {
return nil, err
}
return FromBytes(bs), nil
case nil:
return FromNull(), nil
default:
return nil, errors.New("Unhandled value type")
}
default:
return r.Get(keypath)
}
}
func (e *PrimaryTerm) evalNode(r Record) (res *Value, err error) {
+4 -3
View File
@@ -549,11 +549,12 @@ func timestampCast(v *Value) (t time.Time, _ error) {
func boolCast(v *Value) (b bool, _ error) {
sToB := func(s string) (bool, error) {
if s == "true" {
switch s {
case "true":
return true, nil
} else if s == "false" {
case "false":
return false, nil
} else {
default:
return false, errCastFailure("cannot cast to Bool")
}
}
+84
View File
@@ -0,0 +1,84 @@
{
"title": "Murder on the Orient Express",
"authorInfo": {
"name": "Agatha Christie",
"yearRange": [1890, 1976],
"penName": "Mary Westmacott"
},
"genre": "Crime novel",
"publicationHistory": [
{
"year": 1934,
"publisher": "Collins Crime Club (London)",
"type": "Hardcover",
"pages": 256
},
{
"year": 1934,
"publisher": "Dodd Mead and Company (New York)",
"type": "Hardcover",
"pages": 302
},
{
"year": 2011,
"publisher": "Harper Collins",
"type": "Paperback",
"pages": 265
}
]
}
{
"title": "The Robots of Dawn",
"authorInfo": {
"name": "Isaac Asimov",
"yearRange": [1920, 1992],
"penName": "Paul French"
},
"genre": "Science fiction",
"publicationHistory": [
{
"year": 1983,
"publisher": "Phantasia Press",
"type": "Hardcover",
"pages": 336
},
{
"year": 1984,
"publisher": "Granada",
"type": "Hardcover",
"pages": 419
},
{
"year": 2018,
"publisher": "Harper Voyager",
"type": "Paperback",
"pages": 432
}
]
}
{
"title": "Pigs Have Wings",
"authorInfo": {
"name": "P. G. Wodehouse",
"yearRange": [1881, 1975]
},
"genre": "Comic novel",
"publicationHistory": [
{
"year": 1952,
"publisher": "Doubleday & Company",
"type": "Hardcover"
},
{
"year": 2000,
"publisher": "Harry N. Abrams",
"type": "Hardcover"
},
{
"year": 2019,
"publisher": "Ulverscroft Collections",
"type": "Paperback",
"pages": 294
}
]
}
+100
View File
@@ -0,0 +1,100 @@
/*
* Minio Cloud Storage, (C) 2019 Minio, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package sql
import (
"errors"
"github.com/bcicen/jstream"
)
var (
errKeyLookup = errors.New("Cannot look up key in non-object value")
errIndexLookup = errors.New("Cannot look up array index in non-array value")
errWildcardObjectLookup = errors.New("Object wildcard used on non-object value")
errWildcardArrayLookup = errors.New("Array wildcard used on non-array value")
errWilcardObjectUsageInvalid = errors.New("Invalid usage of object wildcard")
)
func jsonpathEval(p []*JSONPathElement, v interface{}) (r interface{}, err error) {
// fmt.Printf("JPATHexpr: %v jsonobj: %v\n\n", p, v)
if len(p) == 0 || v == nil {
return v, nil
}
switch {
case p[0].Key != nil:
key := p[0].Key.keyString()
kvs, ok := v.(jstream.KVS)
if !ok {
return nil, errKeyLookup
}
for _, kv := range kvs {
if kv.Key == key {
return jsonpathEval(p[1:], kv.Value)
}
}
// Key not found - return nil result
return nil, nil
case p[0].Index != nil:
idx := *p[0].Index
arr, ok := v.([]interface{})
if !ok {
return nil, errIndexLookup
}
if idx >= len(arr) {
return nil, nil
}
return jsonpathEval(p[1:], arr[idx])
case p[0].ObjectWildcard:
kvs, ok := v.(jstream.KVS)
if !ok {
return nil, errWildcardObjectLookup
}
if len(p[1:]) > 0 {
return nil, errWilcardObjectUsageInvalid
}
return kvs, nil
case p[0].ArrayWildcard:
arr, ok := v.([]interface{})
if !ok {
return nil, errWildcardArrayLookup
}
// Lookup remainder of path in each array element and
// make result array.
var result []interface{}
for _, a := range arr {
rval, err := jsonpathEval(p[1:], a)
if err != nil {
return nil, err
}
result = append(result, rval)
}
return result, nil
}
panic("cannot reach here")
}

Some files were not shown because too many files have changed in this diff Show More