mirror of
https://github.com/pgsty/minio.git
synced 2026-08-10 00:03:29 +03:00
Feng Ruohang
fd2ca1c6d2
README, README_ZH, SECURITY, COMPLIANCE, CONTRIBUTING, NOTICE, code_of_conduct, the vulnerability and PR-etiquette documents, the GitHub issue and pull request templates, and the docs/ tree all present Silo as the product. The Grafana dashboards under docs/metrics/prometheus/grafana/ have their panel titles and descriptions rebranded while every minio_* query, label and expression is left alone, so existing alerts and recording rules keep matching. The distinction the review demanded is applied per hit rather than by search-and-replace: - Product and command text becomes Silo and silo: install and run instructions, systemd examples, compose services, download links, badges. - Protocol and interface text keeps MinIO: MINIO_* variables, minio_* metrics, x-minio-* headers, /minio/* routes, .minio.sys, arn:minio, and API field and error names. - Attribution keeps MinIO and gains the fork's own: the AGPL obligations, original copyright, CREDITS and NOTICE stay, with the modification notice added alongside rather than replacing them. - Historical and third-party references are left as facts, not rewritten for brand tidiness. README and README_ZH each carry an explicit non-affiliation notice, document the side-by-side package migration including the /etc/systemd/system/silo.service.d/10-legacy-user.conf drop-in for keeping a legacy UID/GID, and state that recursive chown is never performed. The trademark attribution uses the policy's approved "based on MinIO technology" wording, not the shortened form the policy rejects. github.com/pgsty/minio links are left in place and labelled transitional. The repository has not been renamed, and rewriting them now would produce documented URLs that 404 until the cutover; they change in the cutover commit together with the goreleaser release target, the OCI source label and the raw-content branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
138 lines
5.0 KiB
Markdown
138 lines
5.0 KiB
Markdown
# Compression Guide
|
|
|
|
Silo server allows streaming compression to ensure efficient disk space usage.
|
|
Compression happens inflight, i.e objects are compressed before being written to disk(s).
|
|
Silo uses [`klauspost/compress/s2`](https://github.com/klauspost/compress/tree/master/s2)
|
|
streaming compression due to its stability and performance.
|
|
|
|
This algorithm is specifically optimized for machine generated content.
|
|
Write throughput is typically at least 500MB/s per CPU core,
|
|
and scales with the number of available CPU cores.
|
|
Decompression speed is typically at least 1GB/s.
|
|
|
|
This means that in cases where raw IO is below these numbers
|
|
compression will not only reduce disk usage but also help increase system throughput.
|
|
Typically, enabling compression on spinning disk systems
|
|
will increase speed when the content can be compressed.
|
|
|
|
## Get Started
|
|
|
|
### 1. Prerequisites
|
|
|
|
Install Silo - [Silo Quickstart Guide](https://silo.pgsty.com/operations/deployments/baremetal-deploy-minio-on-redhat-linux/).
|
|
|
|
### 2. Run Silo with compression
|
|
|
|
Compression can be enabled by updating the `compress` config settings for Silo server config.
|
|
Config `compress` settings take extensions and mime-types to be compressed.
|
|
|
|
```bash
|
|
~ mc admin config get mysilo compression
|
|
compression extensions=".txt,.log,.csv,.json,.tar,.xml,.bin" mime_types="text/*,application/json,application/xml"
|
|
```
|
|
|
|
Default config includes most common highly compressible content extensions and mime-types.
|
|
|
|
```bash
|
|
~ mc admin config set mysilo compression extensions=".pdf" mime_types="application/pdf"
|
|
```
|
|
|
|
To show help on setting compression config values.
|
|
|
|
```bash
|
|
~ mc admin config set mysilo compression
|
|
```
|
|
|
|
To enable compression for all content, no matter the extension and content type
|
|
(except for the default excluded types) set BOTH extensions and mime types to empty.
|
|
|
|
```bash
|
|
~ mc admin config set mysilo compression enable="on" extensions="" mime_types=""
|
|
```
|
|
|
|
The compression settings may also be set through environment variables.
|
|
When set, environment variables override the defined `compress` config settings in the server config.
|
|
|
|
```bash
|
|
export MINIO_COMPRESSION_ENABLE="on"
|
|
export MINIO_COMPRESSION_EXTENSIONS=".txt,.log,.csv,.json,.tar,.xml,.bin"
|
|
export MINIO_COMPRESSION_MIME_TYPES="text/*,application/json,application/xml"
|
|
```
|
|
|
|
> [!NOTE]
|
|
> To enable compression for all content when using environment variables, set either or both of the extensions and MIME types to `*` instead of an empty string:
|
|
> ```bash
|
|
> export MINIO_COMPRESSION_ENABLE="on"
|
|
> export MINIO_COMPRESSION_EXTENSIONS="*"
|
|
> export MINIO_COMPRESSION_MIME_TYPES="*"
|
|
> ```
|
|
|
|
### 3. Compression + Encryption
|
|
|
|
Combining encryption and compression is not safe in all setups.
|
|
This is particularly so if the compression ratio of your content reveals information about it.
|
|
See [CRIME TLS](https://en.wikipedia.org/wiki/CRIME) as an example of this.
|
|
|
|
Therefore, compression is disabled when encrypting by default, and must be enabled separately.
|
|
|
|
Evaluate the security and resource impact in a staging environment before
|
|
deciding whether this feature combination is safe for your setup.
|
|
|
|
To enable compression+encryption use:
|
|
|
|
```bash
|
|
~ mc admin config set mysilo compression allow_encryption=on
|
|
```
|
|
|
|
Or alternatively through the environment variable `MINIO_COMPRESSION_ALLOW_ENCRYPTION=on`.
|
|
|
|
### 4. Excluded Types
|
|
|
|
- Already compressed objects are not fit for compression since they do not have compressible patterns.
|
|
Such objects do not produce efficient [`LZ compression`](https://en.wikipedia.org/wiki/LZ77_and_LZ78)
|
|
which is a fitness factor for a lossless data compression.
|
|
|
|
Pre-compressed input typically compresses in excess of 2GiB/s per core,
|
|
so performance impact should be minimal even if precompressed data is re-compressed.
|
|
Decompressing incompressible data has no significant performance impact.
|
|
|
|
Below is a list of common files and content-types which are typically not suitable for compression.
|
|
|
|
- Extensions
|
|
|
|
| `gz` | (GZIP) |
|
|
| `bz2` | (BZIP2) |
|
|
| `rar` | (WinRAR) |
|
|
| `zip` | (ZIP) |
|
|
| `7z` | (7-Zip) |
|
|
| `xz` | (LZMA) |
|
|
| `mp4` | (MP4) |
|
|
| `mkv` | (MKV media) |
|
|
| `mov` | (MOV) |
|
|
|
|
- Content-Types
|
|
|
|
| `video/*` |
|
|
| `audio/*` |
|
|
| `application/zip` |
|
|
| `application/x-gzip` |
|
|
| `application/zip` |
|
|
| `application/x-bz2` |
|
|
| `application/x-compress` |
|
|
| `application/x-xz` |
|
|
|
|
All files with these extensions and mime types are excluded from compression,
|
|
even if compression is enabled for all types.
|
|
|
|
## To test the setup
|
|
|
|
To test this setup, practice put calls to the server using `mc` and use `mc ls` on
|
|
the data directory to view the size of the object.
|
|
|
|
## Explore Further
|
|
|
|
- [Use `mc` with Silo Server](https://silo.pgsty.com/reference/minio-mc/)
|
|
- [Use `aws-cli` with Silo Server](https://silo.pgsty.com/integrations/aws-cli-with-minio/)
|
|
- [Use `minio-go` SDK with Silo Server](https://silo.pgsty.com/developers/go/minio-go/)
|
|
- [The Silo documentation website](https://silo.pgsty.com/docs/)
|