astelm/minio
1
0
Fork 0
mirror of https://github.com/pgsty/minio.git synced 2026-08-09 07:43:29 +03:00
Code Issues Packages Projects Releases Wiki Activity

Performance improvements to SELECT API on certain query operations (#6752)

Browse Source 
This improves the performance of certain queries dramatically,
such as 'count(*)' etc.

Without this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m42.464s
user	0m0.071s
sys	0m0.010s
```

With this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m17.603s
user	0m0.093s
sys	0m0.008s
```

Almost a 250% improvement in performance. This PR avoids a lot of type
conversions and instead relies on raw sequences of data and interprets
them lazily.

```
benchcmp old new
benchmark                        old ns/op       new ns/op       delta
BenchmarkSQLAggregate_100K-4     551213          259782          -52.87%
BenchmarkSQLAggregate_1M-4       6981901985      2432413729      -65.16%
BenchmarkSQLAggregate_2M-4       13511978488     4536903552      -66.42%
BenchmarkSQLAggregate_10M-4      68427084908     23266283336     -66.00%

benchmark                        old allocs     new allocs     delta
BenchmarkSQLAggregate_100K-4     2366           485            -79.50%
BenchmarkSQLAggregate_1M-4       47455492       21462860       -54.77%
BenchmarkSQLAggregate_2M-4       95163637       43110771       -54.70%
BenchmarkSQLAggregate_10M-4      476959550      216906510      -54.52%

benchmark                        old bytes       new bytes      delta
BenchmarkSQLAggregate_100K-4     1233079         1086024        -11.93%
BenchmarkSQLAggregate_1M-4       2607984120      557038536      -78.64%
BenchmarkSQLAggregate_2M-4       5254103616      1128149168     -78.53%
BenchmarkSQLAggregate_10M-4      26443524872     5722715992     -78.36%
```
This commit is contained in:
Harshavardhana
2018-11-14 15:55:10 -08:00
committed by kannappanr
parent f9779b24ad
commit 7e1661f4fa
108 changed files with 640 additions and 12237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vendor/vendor.json
Vendored
+3 -21
View File
@@ -527,12 +527,6 @@
"revision": "7e6a47b300b10be9449610a6ff4fbae17d6e95b6",
"revisionTime": "2018-01-16T16:19:11Z"
},
{
"checksumSHA1": "WqeEgS7pqqkwK8mlrAZmDgtWJMY=",
"path": "github.com/json-iterator/go",
"revision": "1624edc4454b8682399def8740d46db5e4362ba4",
"revisionTime": "2018-08-06T06:07:27Z"
},
{
"checksumSHA1": "mA7isU/nIAT5ytwIzK65H0vlVqI=",
"path": "github.com/klauspost/compress/flate",
@@ -716,18 +710,6 @@
"revision": "281073eb9eb092240d33ef253c404f1cca550309",
"revisionTime": "2015-07-17T05:11:58Z"
},
{
"checksumSHA1": "ZTcgWKWHsrX0RXYVXn5Xeb8Q0go=",
"path": "github.com/modern-go/concurrent",
"revision": "bacd9c7ef1dd9b15be4a9909b8ac7a4e313eec94",
"revisionTime": "2018-03-06T01:26:44Z"
},
{
"checksumSHA1": "qvH48wzTIV3QKSDqI0dLFtVjaDI=",
"path": "github.com/modern-go/reflect2",
"revision": "94122c33edd36123c84d5368cfb2b69df93a0ec8",
"revisionTime": "2018-07-18T01:23:57Z"
},
{
"checksumSHA1": "zvQr4zOz1/g/Fui6co0sctxrJ28=",
"path": "github.com/nats-io/go-nats",
@@ -902,10 +884,10 @@
"revisionTime": "2016-03-11T21:55:03Z"
},
{
"checksumSHA1": "d+4iuDj/qQ+2LiWEbQ+AWjxEJOc=",
"checksumSHA1": "QI+LuP9gKNIfftobaThCcAnOCX0=",
"path": "github.com/tidwall/gjson",
"revision": "f123b340873a0084cb27267eddd8ff615115fbff",
"revisionTime": "2018-06-21T18:09:58Z"
"revision": "081192fa2e471ddff3ec98a3cc04fecfd031ed57",
"revisionTime": "2018-10-28T15:46:04Z"
},
{
"checksumSHA1": "qmePMXEDYGwkAfT9QvtMC58JN/E=",