README, README_ZH, SECURITY, COMPLIANCE, CONTRIBUTING, NOTICE,
code_of_conduct, the vulnerability and PR-etiquette documents, the GitHub issue
and pull request templates, and the docs/ tree all present Silo as the product.
The Grafana dashboards under docs/metrics/prometheus/grafana/ have their panel
titles and descriptions rebranded while every minio_* query, label and
expression is left alone, so existing alerts and recording rules keep matching.
The distinction the review demanded is applied per hit rather than by
search-and-replace:
- Product and command text becomes Silo and silo: install and run instructions,
systemd examples, compose services, download links, badges.
- Protocol and interface text keeps MinIO: MINIO_* variables, minio_* metrics,
x-minio-* headers, /minio/* routes, .minio.sys, arn:minio, and API field and
error names.
- Attribution keeps MinIO and gains the fork's own: the AGPL obligations,
original copyright, CREDITS and NOTICE stay, with the modification notice
added alongside rather than replacing them.
- Historical and third-party references are left as facts, not rewritten for
brand tidiness.
README and README_ZH each carry an explicit non-affiliation notice, document
the side-by-side package migration including the
/etc/systemd/system/silo.service.d/10-legacy-user.conf drop-in for keeping a
legacy UID/GID, and state that recursive chown is never performed. The trademark
attribution uses the policy's approved "based on MinIO technology" wording, not
the shortened form the policy rejects.
github.com/pgsty/minio links are left in place and labelled transitional. The
repository has not been renamed, and rewriting them now would produce documented
URLs that 404 until the cutover; they change in the cutover commit together with
the goreleaser release target, the OCI source label and the raw-content branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# ILM Tiering Design [](https://slack.min.io) [](https://hub.docker.com/r/minio/minio/)
# ILM Tiering Design [](https://hub.docker.com/r/pgsty/silo/)
Lifecycle transition functionality provided in [bucket lifecycle guide](https://github.com/pgsty/minio/blob/master/docs/bucket/lifecycle/README.md) allows tiering of content from MinIO object store to public clouds or other MinIO clusters.
Lifecycle transition functionality provided in [bucket lifecycle guide](https://github.com/pgsty/minio/blob/master/docs/bucket/lifecycle/README.md) allows tiering of content from Silo object store to public clouds or other Silo clusters.
Transition tiers can be added to MinIO using `mc admin tier add` command to associate a `gcs`, `s3` or `azure` bucket or prefix path on a bucket to the tier name.
Transition tiers can be added to Silo using `mc admin tier add` command to associate a `gcs`, `s3` or `azure` bucket or prefix path on a bucket to the tier name.
Lifecycle transition rules can be applied to buckets (both versioned and un-versioned) by specifying the tier name defined above as the transition storage class for the lifecycle rule.
## Implementation
ILM tiering takes place when a object placed in the bucket meets lifecycle transition rules and becomes eligible for tiering. MinIO scanner (which runs at one minute intervals, each time scanning one sixteenth of the namespace), picks up the object for tiering. The data is moved to the remote tier in entirety, leaving only the object metadata on MinIO.
ILM tiering takes place when a object placed in the bucket meets lifecycle transition rules and becomes eligible for tiering. Silo scanner (which runs at one minute intervals, each time scanning one sixteenth of the namespace), picks up the object for tiering. The data is moved to the remote tier in entirety, leaving only the object metadata on Silo.
The data on the backend is stored under the `bucket/prefix` specified in the tier configuration with a custom name derived from a randomly generated uuid - e.g. `0b/c4/0bc4fab7-2daf-4d2f-8e39-5c6c6fb7e2d3`. The first two prefixes are characters 1-2,3-4 from the uuid. This format allows tiering to any cloud irrespective of whether the cloud in question supports versioning. The reference to the transitioned object name and transitioned tier is stored as part of the internal metadata for the object (or its version) on MinIO.
The data on the backend is stored under the `bucket/prefix` specified in the tier configuration with a custom name derived from a randomly generated uuid - e.g. `0b/c4/0bc4fab7-2daf-4d2f-8e39-5c6c6fb7e2d3`. The first two prefixes are characters 1-2,3-4 from the uuid. This format allows tiering to any cloud irrespective of whether the cloud in question supports versioning. The reference to the transitioned object name and transitioned tier is stored as part of the internal metadata for the object (or its version) on Silo.
Extra metadata maintained internally in `xl.meta` for a transitioned object
@@ -22,7 +22,7 @@ Extra metadata maintained internally in `xl.meta` for a transitioned object
},
```
When a transitioned object is restored temporarily to local MinIO instance via PostRestoreObject API, the object data is copied back from the remote tier, and additional metadata for the restored object is maintained as referenced below. Once the restore period expires, the local copy of the object is removed by the scanner during its periodic runs.
When a transitioned object is restored temporarily to local Silo instance via PostRestoreObject API, the object data is copied back from the remote tier, and additional metadata for the restored object is maintained as referenced below. Once the restore period expires, the local copy of the object is removed by the scanner during its periodic runs.
```
...
@@ -35,11 +35,11 @@ When a transitioned object is restored temporarily to local MinIO instance via P
### Encrypted/Object locked objects
For objects under SSE-S3 or SSE-C encryption, the encrypted content from MinIO cluster is copied as is to the remote tier without any decryption. The content is decrypted as it is streamed from remote tier on `GET/HEAD`. Objects under retention are protected because the metadata present on MinIO server ensures that the object (version) is not deleted until retention period is over. Administrators need to ensure that the remote tier bucket is under proper access control.
For objects under SSE-S3 or SSE-C encryption, the encrypted content from Silo cluster is copied as is to the remote tier without any decryption. The content is decrypted as it is streamed from remote tier on `GET/HEAD`. Objects under retention are protected because the metadata present on Silo server ensures that the object (version) is not deleted until retention period is over. Administrators need to ensure that the remote tier bucket is under proper access control.
### Transition Status
MinIO specific extension header `X-Minio-Transition` is displayed on `HEAD/GET` to predict expected transition date on the object. Once object is transitioned to the remote tier,`x-amz-storage-class` shows the tier name to which object transitioned. Additional headers such as "X-Amz-Restore-Expiry-Days", "x-amz-restore", and "X-Amz-Restore-Request-Date" are displayed when a object is under restore/has been restored to local MinIO cluster.
Silo specific extension header `X-Minio-Transition` is displayed on `HEAD/GET` to predict expected transition date on the object. Once object is transitioned to the remote tier,`x-amz-storage-class` shows the tier name to which object transitioned. Additional headers such as "X-Amz-Restore-Expiry-Days", "x-amz-restore", and "X-Amz-Restore-Request-Date" are displayed when a object is under restore/has been restored to local Silo cluster.
### Expiry or removal events
@@ -47,9 +47,9 @@ An object that is in transition tier will be deleted once the object hits expiry
### Additional notes
Tiering and lifecycle transition are applicable only to erasure/distributed MinIO.
Tiering and lifecycle transition are applicable only to erasure/distributed Silo.
## Explore Further
- [MinIO | Golang Client API Reference](https://pkg.go.dev/github.com/minio/minio-go/v7#Client.SetBucketLifecycle)
- [MinIO Go client API reference (S3-compatible SDK)](https://pkg.go.dev/github.com/minio/minio-go/v7#Client.SetBucketLifecycle)
@@ -84,9 +84,9 @@ e.g., To scan objects stored under `user-uploads/` prefix and remove versions ol
}
```
This JSON rule is equivalent to the following MinIO Client command:
This JSON rule is equivalent to the following compatible `mc` command:
```
mc ilm rule add --noncurrent-expire-days 365 --prefix "user-uploads/" myminio/mydata
mc ilm rule add --noncurrent-expire-days 365 --prefix "user-uploads/" mysilo/mydata
```
### 3.2 Automatic removal of noncurrent versions keeping only most recent ones after noncurrent days
@@ -113,14 +113,14 @@ e.g, To remove noncurrent versions of all objects keeping the most recent 5 nonc
}
```
This JSON rule is equivalent to the following MinIO Client command:
This JSON rule is equivalent to the following compatible `mc` command:
```
mc ilm rule add --noncurrent-expire-days 30 --noncurrent-expire-newer 5 --prefix "user-uploads/" myminio/mydata
mc ilm rule add --noncurrent-expire-days 30 --noncurrent-expire-newer 5 --prefix "user-uploads/" mysilo/mydata
```
#### 3.2.a Automatic removal of noncurrent versions keeping only most recent ones immediately (MinIO only extension)
#### 3.2.a Automatic removal of noncurrent versions keeping only most recent ones immediately (Silo only extension)
This is available only on MinIO as an extension to the NewerNoncurrentVersions feature. The following rule makes it possible to remove older noncurrent versions
This is available only on Silo as an extension to the NewerNoncurrentVersions feature. The following rule makes it possible to remove older noncurrent versions
of objects under the prefix `user-uploads/` as soon as there are more than `N` noncurrent versions of an object.
```
@@ -141,9 +141,9 @@ of objects under the prefix `user-uploads/` as soon as there are more than `N` n
```
Note: This rule has an implicit zero NoncurrentDays, which makes the expiry of those 'extra' noncurrent versions immediate.
#### 3.2.b Automatic removal of all versions (MinIO only extension)
#### 3.2.b Automatic removal of all versions (Silo only extension)
This is available only on MinIO as an extension to the Expiration feature. The following rule makes it possible to remove all versions of an object under
This is available only on Silo as an extension to the Expiration feature. The following rule makes it possible to remove all versions of an object under
the prefix `user-uploads/` as soon as the latest object satisfies the expiration criteria.
> NOTE: If the latest object is a delete marker then filtering based on `Filter.Tags` is ignored and
@@ -188,9 +188,9 @@ When an object has only one version as a delete marker, the latter can be automa
## 4. Enable ILM transition feature
In Erasure mode, MinIO supports tiering to public cloud providers such as GCS, AWS and Azure as well as to other MinIO clusters via the ILM transition feature. This will allow transitioning of older objects to a different cluster or the public cloud by setting up transition rules in the bucket lifecycle configuration. This feature enables applications to optimize storage costs by moving less frequently accessed data to a cheaper storage without compromising accessibility of data.
In Erasure mode, Silo supports tiering to public cloud providers such as GCS, AWS and Azure as well as to other Silo clusters via the ILM transition feature. This will allow transitioning of older objects to a different cluster or the public cloud by setting up transition rules in the bucket lifecycle configuration. This feature enables applications to optimize storage costs by moving less frequently accessed data to a cheaper storage without compromising accessibility of data.
To transition objects in a bucket to a destination bucket on a different cluster, applications need to specify a transition tier defined on MinIO instead of storage class while setting up the ILM lifecycle rule.
To transition objects in a bucket to a destination bucket on a different cluster, applications need to specify a transition tier defined on Silo instead of storage class while setting up the ILM lifecycle rule.
> To create a transition tier for transitioning objects to a prefix `testprefix` in `azurebucket` on Azure blob using `mc`:
@@ -203,10 +203,10 @@ To transition objects in a bucket to a destination bucket on a different cluster
Using above tier, set up a lifecycle rule with transition:
```
mc ilm add --expiry-days 365 --transition-days 45 --storage-class "AZURETIER" myminio/srcbucket
mc ilm add --expiry-days 365 --transition-days 45 --storage-class "AZURETIER" mysilo/srcbucket
```
Note: In the case of S3, it is possible to create a tier from MinIO running in EC2 to S3 using AWS role attached to EC2 as credentials instead of accesskey/secretkey:
Note: In the case of S3, it is possible to create a tier from Silo running in EC2 to S3 using AWS role attached to EC2 as credentials instead of accesskey/secretkey:
`s3:ObjectTransition:Complete` and `s3:ObjectTransition:Failed` events can be used to monitor transition events between the source cluster and transition tier. To watch lifecycle events, you can enable bucket notification on the source bucket with `mc event add` and specify `--event ilm` flag.
Note that transition event notification is a MinIO extension.
Note that transition event notification is a Silo extension.
## Explore Further
- [MinIO | Golang Client API Reference](https://pkg.go.dev/github.com/minio/minio-go/v7)
- [MinIO Go client API reference (S3-compatible SDK)](https://pkg.go.dev/github.com/minio/minio-go/v7)
@@ -30,7 +30,11 @@ Various event types supported by MinIO server are
| `s3:BucketCreated` |
| `s3:BucketRemoved` |
Use client tools like `mc` to set and listen for event notifications using the [`event` sub-command](https://silo.pgsty.com/reference/minio-mc/mc-event-add/). MinIO SDK's [`BucketNotification` APIs](https://pkg.go.dev/github.com/minio/minio-go/v7#Client.SetBucketNotification) can also be used. The notification message MinIO sends to publish an event is a JSON message with the following [structure](https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html).
Use compatible client tools like `mc` to set and listen for event notifications
with the [`event` sub-command](https://silo.pgsty.com/reference/minio-mc/mc-event-add/).
The technical [`minio-go` BucketNotification API](https://pkg.go.dev/github.com/minio/minio-go/v7#Client.SetBucketNotification)
can also be used. Silo publishes the standard S3-compatible JSON event
> - '\*' at the end of the values, means its the default value for the arg.
> - When configured using environment variables, the `:name` can be specified using this format `MINIO_NOTIFY_WEBHOOK_ENABLE_<name>`.
## Publish MinIO events via AMQP
## Publish Silo events via AMQP
Install RabbitMQ from [here](https://www.rabbitmq.com/).
### Step 1: Add AMQP endpoint to MinIO
### Step 1: Add AMQP endpoint to Silo
The AMQP configuration is located under the sub-system `notify_amqp` top-level key. Create a configuration key-value pair here for your AMQP instance. The key is a name for your AMQP endpoint, and the value is a collection of key-value parameters described in the table below.
@@ -117,37 +121,37 @@ MINIO_NOTIFY_AMQP_QUEUE_LIMIT (number) maximum limit for undelivered messa
MINIO_NOTIFY_AMQP_COMMENT (sentence) optionally add a comment to this setting
```
MinIO supports persistent event store. The persistent store will backup events when the AMQP broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the AMQP broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
To update the configuration, use `mc admin config get notify_amqp` command to get the current configuration for `notify_amqp`.
Use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment.Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
An example configuration for RabbitMQ is shown below:
```sh
mc admin config set myminio/ notify_amqp:1 exchange="bucketevents"exchange_type="fanout"mandatory="off"no_wait="off"url="amqp://myuser:mypassword@localhost:5672"auto_deleted="off"delivery_mode="0"durable="off"internal="off"routing_key="bucketlogs"
mc admin config set mysilo/ notify_amqp:1 exchange="bucketevents"exchange_type="fanout"mandatory="off"no_wait="off"url="amqp://myuser:mypassword@localhost:5672"auto_deleted="off"delivery_mode="0"durable="off"internal="off"routing_key="bucketlogs"
```
MinIO supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using `fanout` exchange.
Silo supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using `fanout` exchange.
MinIO also sends with the notifications two headers: `minio-bucket` and `minio-event`. An exchange using the type "headers" can use this information to route the notifications to proper queues.
Silo also sends with the notifications two headers: `minio-bucket` and `minio-event`. An exchange using the type "headers" can use this information to route the notifications to proper queues.
Note that, you can add as many AMQP server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the AMQP instance and an object of per-server configuration parameters.
### Step 2: Enable RabbitMQ bucket notification using MinIO client
### Step 2: Enable RabbitMQ bucket notification using Silo client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:amqp`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `mysilo` server. Here ARN value is `arn:minio:sqs::1:amqp`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:amqp --suffix .jpg
mc event list myminio/images
mc mb mysilo/images
mc event add mysilo/images arn:minio:sqs::1:amqp --suffix .jpg
Install an MQTT Broker from [here](https://mosquitto.org/).
### Step 1: Add MQTT endpoint to MinIO
### Step 1: Add MQTT endpoint to Silo
The MQTT configuration is located as `notify_mqtt` key. Create a configuration key-value pair here for your MQTT instance. The key is a name for your MQTT endpoint, and the value is a collection of key-value parameters described in the table below.
@@ -248,33 +252,33 @@ MINIO_NOTIFY_MQTT_QUEUE_LIMIT (number) maximum limit for undelivered
MINIO_NOTIFY_MQTT_COMMENT (sentence) optionally add a comment to this setting
```
MinIO supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
To update the configuration, use `mc admin config get` command to get the current configuration.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
```sh
mc admin config set myminio notify_mqtt:1 broker="tcp://localhost:1883"password=""queue_dir=""queue_limit="0"reconnect_interval="0s"keep_alive_interval="0s"qos="1"topic="minio"username=""
mc admin config set mysilo notify_mqtt:1 broker="tcp://localhost:1883"password=""queue_dir=""queue_limit="0"reconnect_interval="0s"keep_alive_interval="0s"qos="1"topic="minio"username=""
```
MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using `tcp://`, `tls://`, or `ws://` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
Silo supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using `tcp://`, `tls://`, or `ws://` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
Note that, you can add as many MQTT server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MQTT instance and an object of per-server configuration parameters.
### Step 2: Enable MQTT bucket notification using MinIO client
### Step 2: Enable MQTT bucket notification using Silo client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:mqtt`.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `mysilo` server. Here ARN value is `arn:minio:sqs::1:mqtt`.
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:mqtt --suffix .jpg
mc event list myminio/images
mc mb mysilo/images
mc event add mysilo/images arn:minio:sqs::1:mqtt --suffix .jpg
This notification target supports two formats: _namespace_ and _access_.
When the _namespace_ format is used, MinIO synchronizes objects in the bucket with documents in the index. For each event in the MinIO, the server creates a document with the bucket and object name from the event as the document ID. Other details of the event are stored in the body of the document. Thus if an existing object is over-written in MinIO, the corresponding document in the Elasticsearch index is updated. If an object is deleted, the corresponding document is deleted from the index.
When the _namespace_ format is used, Silo synchronizes objects in the bucket with documents in the index. For each event in the Silo, the server creates a document with the bucket and object name from the event as the document ID. Other details of the event are stored in the body of the document. Thus if an existing object is over-written in Silo, the corresponding document in the Elasticsearch index is updated. If an object is deleted, the corresponding document is deleted from the index.
When the _access_ format is used, MinIO appends events as documents in an Elasticsearch index. For each event, a document with the event details, with the timestamp of document set to the event's timestamp is appended to an index. The ID of the documented is randomly generated by Elasticsearch. No documents are deleted or modified in this format.
When the _access_ format is used, Silo appends events as documents in an Elasticsearch index. For each event, a document with the event details, with the timestamp of document set to the event's timestamp is appended to an index. The ID of the documented is randomly generated by Elasticsearch. No documents are deleted or modified in this format.
The steps below show how to use this notification target in `namespace` format. The other format is very similar and is omitted for brevity.
### Step 1: Ensure Elasticsearch minimum requirements are met
MinIO requires a 5.x series version of Elasticsearch. This is the latest major release series. Elasticsearch provides version upgrade migration guidelines [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html).
Silo requires a 5.x series version of Elasticsearch. This is the latest major release series. Elasticsearch provides version upgrade migration guidelines [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html).
### Step 2: Add Elasticsearch endpoint to MinIO
### Step 2: Add Elasticsearch endpoint to Silo
The Elasticsearch configuration is located in the `notify_elasticsearch` key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.
@@ -381,37 +385,37 @@ MINIO_NOTIFY_ELASTICSEARCH_COMMENT (sentence) optionally add a co
For example: `http://localhost:9200` or with authentication info `http://elastic:MagicWord@127.0.0.1:9200`.
MinIO supports persistent event store. The persistent store will backup events when the Elasticsearch broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the Elasticsearch broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
If Elasticsearch has authentication enabled, the credentials can be supplied to MinIO via the `url` parameter formatted as `PROTO://USERNAME:PASSWORD@ELASTICSEARCH_HOST:PORT`.
If Elasticsearch has authentication enabled, the credentials can be supplied to Silo via the `url` parameter formatted as `PROTO://USERNAME:PASSWORD@ELASTICSEARCH_HOST:PORT`.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ notify_elasticsearch
$ mc admin config get mysilo/ notify_elasticsearch
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
```sh
mc admin config set myminio notify_elasticsearch:1 queue_limit="0"url="http://127.0.0.1:9200"format="namespace"index="minio_events"queue_dir=""username=""password=""
mc admin config set mysilo notify_elasticsearch:1 queue_limit="0"url="http://127.0.0.1:9200"format="namespace"index="minio_events"queue_dir=""username=""password=""
```
Note that, you can add as many Elasticsearch server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Elasticsearch instance and an object of per-server configuration parameters.
### Step 3: Enable Elastichsearch bucket notification using MinIO client
### Step 3: Enable Elastichsearch bucket notification using Silo client
We will now enable bucket event notifications on a bucket named `images`. Whenever a JPEG image is created/overwritten, a new document is added or an existing document is updated in the Elasticsearch index configured above. When an existing object is deleted, the corresponding document is deleted from the index. Thus, the rows in the Elasticsearch index, reflect the `.jpg` objects in the `images` bucket.
To configure this bucket notification, we need the ARN printed by MinIO in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
To configure this bucket notification, we need the ARN printed by Silo in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
With the `mc` tool, the configuration is very simple to add. Let us say that the MinIO server is aliased as `myminio` in our mc configuration. Execute the following:
With the `mc` tool, the configuration is very simple to add. Let us say that the Silo server is aliased as `mysilo` in our mc configuration. Execute the following:
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:elasticsearch --suffix .jpg
mc event list myminio/images
mc mb mysilo/images
mc event add mysilo/images arn:minio:sqs::1:elasticsearch --suffix .jpg
@@ -497,21 +501,21 @@ This output shows that a document has been created for the event in Elasticsearc
Here we see that the document ID is the bucket and object name. In case `access` format was used, the document ID would be automatically generated by Elasticsearch.
## Publish MinIO events via Redis
## Publish Silo events via Redis
Install [Redis](http://redis.io/download) server. For illustrative purposes, we have set the database password as "yoursecret".
This notification target supports two formats: _namespace_ and _access_.
When the _namespace_ format is used, MinIO synchronizes objects in the bucket with entries in a hash. For each entry, the key is formatted as "bucketName/objectName" for an object that exists in the bucket, and the value is the JSON-encoded event data about the operation that created/replaced the object in MinIO. When objects are updated or deleted, the corresponding entry in the hash is also updated or deleted.
When the _namespace_ format is used, Silo synchronizes objects in the bucket with entries in a hash. For each entry, the key is formatted as "bucketName/objectName" for an object that exists in the bucket, and the value is the JSON-encoded event data about the operation that created/replaced the object in Silo. When objects are updated or deleted, the corresponding entry in the hash is also updated or deleted.
When the _access_ format is used, MinIO appends events to a list using [RPUSH](https://redis.io/commands/rpush). Each item in the list is a JSON encoded list with two items, where the first item is a timestamp string, and the second item is a JSON object containing event data about the operation that happened in the bucket. No entries appended to the list are updated or deleted by MinIO in this format.
When the _access_ format is used, Silo appends events to a list using [RPUSH](https://redis.io/commands/rpush). Each item in the list is a JSON encoded list with two items, where the first item is a timestamp string, and the second item is a JSON object containing event data about the operation that happened in the bucket. No entries appended to the list are updated or deleted by Silo in this format.
The steps below show how to use this notification target in `namespace` and `access` format.
### Step 1: Add Redis endpoint to MinIO
### Step 1: Add Redis endpoint to Silo
The MinIO server configuration file is stored on the backend in json format.The Redis configuration is located in the `redis` key under the `notify` top-level key. Create a configuration key-value pair here for your Redis instance. The key is a name for your Redis endpoint, and the value is a collection of key-value parameters described in the table below.
The Silo server configuration file is stored on the backend in json format.The Redis configuration is located in the `redis` key under the `notify` top-level key. Create a configuration key-value pair here for your Redis instance. The key is a name for your Redis endpoint, and the value is a collection of key-value parameters described in the table below.
```
KEY:
@@ -543,35 +547,35 @@ MINIO_NOTIFY_REDIS_QUEUE_LIMIT (number) maximum limit for undeliver
MINIO_NOTIFY_REDIS_COMMENT (sentence) optionally add a comment to this setting
```
MinIO supports persistent event store. The persistent store will backup events when the Redis broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the Redis broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
To update the configuration, use `mc admin config get` command to get the current configuration.
Use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment.Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
```sh
mc admin config set myminio/ notify_redis:1 address="127.0.0.1:6379"format="namespace"key="bucketevents"password="yoursecret"queue_dir=""queue_limit="0"
mc admin config set mysilo/ notify_redis:1 address="127.0.0.1:6379"format="namespace"key="bucketevents"password="yoursecret"queue_dir=""queue_limit="0"
```
Note that, you can add as many Redis server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Redis instance and an object of per-server configuration parameters.
### Step 2: Enable Redis bucket notification using MinIO client
### Step 2: Enable Redis bucket notification using Silo client
We will now enable bucket event notifications on a bucket named `images`. Whenever a JPEG image is created/overwritten, a new key is added or an existing key is updated in the Redis hash configured above. When an existing object is deleted, the corresponding key is deleted from the Redis hash. Thus, the rows in the Redis hash, reflect the `.jpg` objects in the `images` bucket.
To configure this bucket notification, we need the ARN printed by MinIO in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
To configure this bucket notification, we need the ARN printed by Silo in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
With the `mc` tool, the configuration is very simple to add. Let us say that the MinIO server is aliased as `myminio` in our mc configuration. Execute the following:
With the `mc` tool, the configuration is very simple to add. Let us say that the Silo server is aliased as `mysilo` in our mc configuration. Execute the following:
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:redis --suffix .jpg
mc event list myminio/images
mc mb mysilo/images
mc event add mysilo/images arn:minio:sqs::1:redis --suffix .jpg
Here we see that MinIO performed `HSET` on `minio_events` key.
Here we see that Silo performed `HSET` on `minio_events` key.
In case, `access` format was used, then `minio_events` would be a list, and the MinIO server would have performed an `RPUSH` to append to the list. A consumer of this list would ideally use `BLPOP` to remove list items from the left-end of the list.
In case, `access` format was used, then `minio_events` would be a list, and the Silo server would have performed an `RPUSH` to append to the list. A consumer of this list would ideally use `BLPOP` to remove list items from the left-end of the list.
## Publish MinIO events via NATS
## Publish Silo events via NATS
Install NATS from [here](http://nats.io/).
### Step 1: Add NATS endpoint to MinIO
### Step 1: Add NATS endpoint to Silo
MinIO supports persistent event store. The persistent store will backup events when the NATS broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the NATS broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
```
KEY:
@@ -665,31 +669,32 @@ MINIO_NOTIFY_NATS_QUEUE_LIMIT (number) maximum limit fo
MINIO_NOTIFY_NATS_COMMENT (sentence) optionally add a comment to this setting
```
To update the configuration, use `mc admin config get`command to get the current configuration file for the minio deployment.
To update the configuration, use `mc admin config get`to retrieve the current
Use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. `bucketevents` is the subject used by NATS in this example.
Use `mc admin config set` command to update the configuration for the deployment.Restart Silo server to reflect config changes. `bucketevents` is the subject used by NATS in this example.
```sh
mc admin config set myminio notify_nats:1 password="yoursecret"streaming_max_pub_acks_in_flight="10"subject=""address="0.0.0.0:4222"token=""username="yourusername"ping_interval="0"queue_limit="0"tls="off"streaming_async="on"queue_dir=""streaming_cluster_id="test-cluster"streaming_enable="on"
mc admin config set mysilo notify_nats:1 password="yoursecret"streaming_max_pub_acks_in_flight="10"subject=""address="0.0.0.0:4222"token=""username="yourusername"ping_interval="0"queue_limit="0"tls="off"streaming_async="on"queue_dir=""streaming_cluster_id="test-cluster"streaming_enable="on"
```
MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/streaming/nats-streaming-intro/) that offers additional functionality like `At-least-once-delivery`, and `Publisher rate limiting`. To configure MinIO server to send notifications to NATS Streaming server, update the MinIO server configuration file as follows:
Silo server also supports [NATS Streaming mode](http://nats.io/documentation/streaming/nats-streaming-intro/) that offers additional functionality like `At-least-once-delivery`, and `Publisher rate limiting`. To configure Silo server to send notifications to NATS Streaming server, update the Silo server configuration file as follows:
Read more about sections `cluster_id`, `client_id` on [NATS documentation](https://github.com/nats-io/nats-streaming-server/blob/master/README.md). Section `maxPubAcksInflight` is explained [here](https://github.com/nats-io/stan.go#publisher-rate-limiting).
### Step 2: Enable NATS bucket notification using MinIO client
### Step 2: Enable NATS bucket notification using Silo client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nats`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `mysilo` server. Here ARN value is `arn:minio:sqs::1:nats`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:nats --suffix .jpg
mc event list myminio/images
mc mb mysilo/images
mc event add mysilo/images arn:minio:sqs::1:nats --suffix .jpg
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg mysilo/images
```
The example `nats.go` program prints event notification to console.
@@ -814,16 +819,16 @@ go run nats.go
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg mysilo/images
```
The example `nats.go` program prints event notification to console.
```
Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2017-07-07T18:46:37Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"192.168.1.80:55328"},"responseElements":{"x-amz-request-id":"14CF20BD1EFD5B93","x-minio-origin-endpoint":"http://127.0.0.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":248682,"eTag":"f1671feacb8bbf7b0397c6e9364e8c92","contentType":"image/jpeg","userDefined":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"14CF20BD1EFD5B93"}},"source":{"host":"192.168.1.80","port":"55328","userAgent":"MinIO (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET"}}],"level":"info","msg":"","time":"2017-07-07T11:46:37-07:00"}
Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2017-07-07T18:46:37Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"192.168.1.80:55328"},"responseElements":{"x-amz-request-id":"14CF20BD1EFD5B93","x-minio-origin-endpoint":"http://127.0.0.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":248682,"eTag":"f1671feacb8bbf7b0397c6e9364e8c92","contentType":"image/jpeg","userDefined":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"14CF20BD1EFD5B93"}},"source":{"host":"192.168.1.80","port":"55328","userAgent":"Silo (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET"}}],"level":"info","msg":"","time":"2017-07-07T11:46:37-07:00"}
```
## Publish MinIO events via PostgreSQL
## Publish Silo events via PostgreSQL
> NOTE: Until release RELEASE.2020-04-10T03-34-42Z PostgreSQL notification used to support following options:
>
@@ -840,28 +845,28 @@ Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jp
> following command to update the existing notification targets.
>
> ```
> mc admin config set myminio/ notify_postgres[:name] connection_string="host=hostname port=2832 username=psqluser password=psqlpass database=bucketevents"
> mc admin config set mysilo/ notify_postgres[:name] connection_string="host=hostname port=2832 username=psqluser password=psqlpass database=bucketevents"
> ```
>
> Please make sure this step is carried out, without this step PostgreSQL notification targets will not work,
> an error message will be shown on the console upon server upgrade/restart, make sure to follow the above
> instructions appropriately. For further questions please join our <https://slack.min.io>
> instructions appropriately. For further guidance, use the [Silo documentation](https://silo.pgsty.com/docs/).
Install [PostgreSQL](https://www.postgresql.org/) database server. For illustrative purposes, we have set the "postgres" user password as `password` and created a database called `minio_events` to store the events.
This notification target supports two formats: _namespace_ and _access_.
When the _namespace_ format is used, MinIO synchronizes objects in the bucket with rows in the table. It creates rows with two columns: key and value. The key is the bucket and object name of an object that exists in MinIO. The value is JSON encoded event data about the operation that created/replaced the object in MinIO. When objects are updated or deleted, the corresponding row from this table is updated or deleted respectively.
When the _namespace_ format is used, Silo synchronizes objects in the bucket with rows in the table. It creates rows with two columns: key and value. The key is the bucket and object name of an object that exists in Silo. The value is JSON encoded event data about the operation that created/replaced the object in Silo. When objects are updated or deleted, the corresponding row from this table is updated or deleted respectively.
When the _access_ format is used, MinIO appends events to a table. It creates rows with two columns: event_time and event_data. The event_time is the time at which the event occurred in the MinIO server. The event_data is the JSON encoded event data about the operation on an object. No rows are deleted or modified in this format.
When the _access_ format is used, Silo appends events to a table. It creates rows with two columns: event_time and event_data. The event_time is the time at which the event occurred in the Silo server. The event_data is the JSON encoded event data about the operation on an object. No rows are deleted or modified in this format.
The steps below show how to use this notification target in `namespace` format. The other format is very similar and is omitted for brevity.
### Step 1: Ensure postgresql minimum requirements are met
MinIO requires PostgreSQL version 9.5 or above. MinIO uses the [`INSERT ON CONFLICT`](https://www.postgresql.org/docs/9.5/static/sql-insert.html#SQL-ON-CONFLICT) (aka UPSERT) feature, introduced in version 9.5 and the [JSONB](https://www.postgresql.org/docs/9.4/static/datatype-json.html) data-type introduced in version 9.4.
Silo requires PostgreSQL version 9.5 or above. Silo uses the [`INSERT ON CONFLICT`](https://www.postgresql.org/docs/9.5/static/sql-insert.html#SQL-ON-CONFLICT) (aka UPSERT) feature, introduced in version 9.5 and the [JSONB](https://www.postgresql.org/docs/9.4/static/datatype-json.html) data-type introduced in version 9.4.
### Step 2: Add PostgreSQL endpoint to MinIO
### Step 2: Add PostgreSQL endpoint to Silo
The PostgreSQL configuration is located in the `notify_postgresql` key. Create a configuration key-value pair here for your PostgreSQL instance. The key is a name for your PostgreSQL endpoint, and the value is a collection of key-value parameters described in the table below.
@@ -899,40 +904,40 @@ MINIO_NOTIFY_POSTGRES_MAX_OPEN_CONNECTIONS (number) maximum number o
> NOTE: If the `max_open_connections` key or the environment variable `MINIO_NOTIFY_POSTGRES_MAX_OPEN_CONNECTIONS` is set to `0`, There will be no limit set on the number of
> open connections to the database. This setting is generally NOT recommended as the behavior may be inconsistent during recursive deletes in `namespace` format.
MinIO supports persistent event store. The persistent store will backup events when the PostgreSQL connection goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the PostgreSQL connection goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Note that for illustration here, we have disabled SSL. In the interest of security, for production this is not recommended.
To update the configuration, use `mc admin config get` command to get the current configuration.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
```sh
mc admin config set myminio notify_postgres:1 connection_string="host=localhost port=5432 dbname=minio_events user=postgres password=password sslmode=disable"table="bucketevents"format="namespace"
mc admin config set mysilo notify_postgres:1 connection_string="host=localhost port=5432 dbname=minio_events user=postgres password=password sslmode=disable"table="bucketevents"format="namespace"
```
Note that, you can add as many PostgreSQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the PostgreSQL instance and an object of per-server configuration parameters.
### Step 3: Enable PostgreSQL bucket notification using MinIO client
### Step 3: Enable PostgreSQL bucket notification using Silo client
We will now enable bucket event notifications on a bucket named `images`. Whenever a JPEG image is created/overwritten, a new row is added or an existing row is updated in the PostgreSQL configured above. When an existing object is deleted, the corresponding row is deleted from the PostgreSQL table. Thus, the rows in the PostgreSQL table, reflect the `.jpg` objects in the `images` bucket.
To configure this bucket notification, we need the ARN printed by MinIO in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
To configure this bucket notification, we need the ARN printed by Silo in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
With the `mc` tool, the configuration is very simple to add. Let us say that the MinIO server is aliased as `myminio` in our mc configuration. Execute the following:
With the `mc` tool, the configuration is very simple to add. Let us say that the Silo server is aliased as `mysilo` in our mc configuration. Execute the following:
```
# Create bucket named `images` in myminio
mc mb myminio/images
# Create bucket named `images` in mysilo
mc mb mysilo/images
# Add notification configuration on the `images` bucket using the MySQL ARN. The --suffix argument filters events.
mc event add myminio/images arn:minio:sqs::1:postgresql --suffix .jpg
mc event add mysilo/images arn:minio:sqs::1:postgresql --suffix .jpg
# Print out the notification configuration on the `images` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg mysilo/images
```
Open PostgreSQL terminal to list the rows in the `bucketevents` table.
@@ -956,7 +961,7 @@ key | value
(1 row)
```
## Publish MinIO events via MySQL
## Publish Silo events via MySQL
> NOTE: Until release RELEASE.2020-04-10T03-34-42Z MySQL notification used to support following options:
>
@@ -973,28 +978,28 @@ key | value
> following command to update the existing notification targets.
>
> ```
> mc admin config set myminio/ notify_mysql[:name] dsn_string="mysqluser:mysqlpass@tcp(localhost:2832)/bucketevents"
> mc admin config set mysilo/ notify_mysql[:name] dsn_string="mysqluser:mysqlpass@tcp(localhost:2832)/bucketevents"
> ```
>
> Please make sure this step is carried out, without this step MySQL notification targets will not work,
> an error message will be shown on the console upon server upgrade/restart, make sure to follow the above
> instructions appropriately. For further questions please join our <https://slack.min.io>
> instructions appropriately. For further guidance, use the [Silo documentation](https://silo.pgsty.com/docs/).
Install MySQL from [here](https://dev.mysql.com/downloads/mysql/). For illustrative purposes, we have set the root password as `password` and created a database called `miniodb` to store the events.
This notification target supports two formats: _namespace_ and _access_.
When the _namespace_ format is used, MinIO synchronizes objects in the bucket with rows in the table. It creates rows with two columns: key_name and value. The key_name is the bucket and object name of an object that exists in MinIO. The value is JSON encoded event data about the operation that created/replaced the object in MinIO. When objects are updated or deleted, the corresponding row from this table is updated or deleted respectively.
When the _namespace_ format is used, Silo synchronizes objects in the bucket with rows in the table. It creates rows with two columns: key_name and value. The key_name is the bucket and object name of an object that exists in Silo. The value is JSON encoded event data about the operation that created/replaced the object in Silo. When objects are updated or deleted, the corresponding row from this table is updated or deleted respectively.
When the _access_ format is used, MinIO appends events to a table. It creates rows with two columns: event_time and event_data. The event_time is the time at which the event occurred in the MinIO server. The event_data is the JSON encoded event data about the operation on an object. No rows are deleted or modified in this format.
When the _access_ format is used, Silo appends events to a table. It creates rows with two columns: event_time and event_data. The event_time is the time at which the event occurred in the Silo server. The event_data is the JSON encoded event data about the operation on an object. No rows are deleted or modified in this format.
The steps below show how to use this notification target in `namespace` format. The other format is very similar and is omitted for brevity.
### Step 1: Ensure MySQL minimum requirements are met
MinIO requires MySQL version 5.7.8 or above. MinIO uses the [JSON](https://dev.mysql.com/doc/refman/5.7/en/json.html) data-type introduced in version 5.7.8. We tested this setup on MySQL 5.7.17.
Silo requires MySQL version 5.7.8 or above. Silo uses the [JSON](https://dev.mysql.com/doc/refman/5.7/en/json.html) data-type introduced in version 5.7.8. We tested this setup on MySQL 5.7.17.
### Step 2: Add MySQL server endpoint configuration to MinIO
### Step 2: Add MySQL server endpoint configuration to Silo
The MySQL configuration is located in the `notify_mysql` key. Create a configuration key-value pair here for your MySQL instance. The key is a name for your MySQL endpoint, and the value is a collection of key-value parameters described in the table below.
@@ -1034,40 +1039,40 @@ MINIO_NOTIFY_MYSQL_COMMENT (sentence) optionally add a co
`dsn_string` is required and is of form `"<user>:<password>@tcp(<host>:<port>)/<database>"`
MinIO supports persistent event store. The persistent store will backup events if MySQL connection goes offline and then replays the stored events when the broken connection comes back up. The event store can be configured by setting a directory path in `queue_dir` field, and the maximum number of events, which can be stored in a `queue_dir`, in `queue_limit` field. For example, `queue_dir` can be set to `/home/events` and `queue_limit` can be set to `1000`. By default, the `queue_limit` is set to `100000`.
Silo supports persistent event store. The persistent store will backup events if MySQL connection goes offline and then replays the stored events when the broken connection comes back up. The event store can be configured by setting a directory path in `queue_dir` field, and the maximum number of events, which can be stored in a `queue_dir`, in `queue_limit` field. For example, `queue_dir` can be set to `/home/events` and `queue_limit` can be set to `1000`. By default, the `queue_limit` is set to `100000`.
Before updating the configuration, let's start with `mc admin config get` command to get the current configuration.
Use `mc admin config set` command to update MySQL notification configuration for the deployment with `dsn_string` parameter:
```sh
mc admin config set myminio notify_mysql:myinstance table="minio_images"dsn_string="root:xxxx@tcp(172.17.0.1:3306)/miniodb"
mc admin config set mysilo notify_mysql:myinstance table="minio_images"dsn_string="root:xxxx@tcp(172.17.0.1:3306)/miniodb"
```
Note that, you can add as many MySQL server endpoint configurations as needed by providing an identifier (like "myinstance" in the example above) for each MySQL instance desired.
Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::myinstance:mysql` at start-up, if there are no errors.
Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::myinstance:mysql` at start-up, if there are no errors.
### Step 3: Enable MySQL bucket notification using MinIO client
### Step 3: Enable MySQL bucket notification using Silo client
We will now setup bucket notifications on a bucket named `images`. Whenever a JPEG image object is created/overwritten, a new row is added or an existing row is updated in the MySQL table configured above. When an existing object is deleted, the corresponding row is deleted from the MySQL table. Thus, the rows in the MySQL table, reflect the `.jpg` objects in the `images` bucket.
To configure this bucket notification, we need the ARN printed by MinIO in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
To configure this bucket notification, we need the ARN printed by Silo in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
With the `mc` tool, the configuration is very simple to add. Let us say that the MinIO server is aliased as `myminio` in our mc configuration. Execute the following:
With the `mc` tool, the configuration is very simple to add. Let us say that the Silo server is aliased as `mysilo` in our mc configuration. Execute the following:
```
# Create bucket named `images` in myminio
mc mb myminio/images
# Create bucket named `images` in mysilo
mc mb mysilo/images
# Add notification configuration on the `images` bucket using the MySQL ARN. The --suffix argument filters events.
mc event add myminio/images arn:minio:sqs::myinstance:mysql --suffix .jpg
mc event add mysilo/images arn:minio:sqs::myinstance:mysql --suffix .jpg
# Print out the notification configuration on the `images` bucket.
Open another terminal and upload a JPEG image into `images` bucket:
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg mysilo/images
```
Open MySQL terminal and list the rows in the `minio_images` table.
@@ -1093,17 +1098,17 @@ mysql> select * from minio_images;
```
## Publish MinIO events via Kafka
## Publish Silo events via Kafka
Install Apache Kafka from [here](http://kafka.apache.org/).
### Step 1: Ensure minimum requirements are met
MinIO requires Kafka version 0.10 or 0.9. Internally MinIO uses the [Shopify/sarama](https://github.com/Shopify/sarama/) library and so has the same version compatibility as provided by this library.
Silo requires Kafka version 0.10 or 0.9. Internally Silo uses the [Shopify/sarama](https://github.com/Shopify/sarama/) library and so has the same version compatibility as provided by this library.
### Step 2: Add Kafka endpoint to MinIO
### Step 2: Add Kafka endpoint to Silo
MinIO supports persistent event store. The persistent store will backup events when the kafka broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the kafka broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.`bucketevents` is the topic used by kafka in this example.
Use `mc admin config set` command to update the configuration for the deployment. Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.`bucketevents` is the topic used by kafka in this example.
```sh
mc admin config set myminio notify_kafka:1 tls_skip_verify="off"queue_dir=""queue_limit="0"sasl="off"sasl_password=""sasl_username=""tls_client_auth="0"tls="off"client_tls_cert=""client_tls_key=""brokers="localhost:9092,localhost:9093"topic="bucketevents"version=""
mc admin config set mysilo notify_kafka:1 tls_skip_verify="off"queue_dir=""queue_limit="0"sasl="off"sasl_password=""sasl_username=""tls_client_auth="0"tls="off"client_tls_cert=""client_tls_key=""brokers="localhost:9092,localhost:9093"topic="bucketevents"version=""
```
### Step 3: Enable Kafka bucket notification using MinIO client
### Step 3: Enable Kafka bucket notification using Silo client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:kafka`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `mysilo` server. Here ARN value is `arn:minio:sqs::1:kafka`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:kafka --suffix .jpg
mc event list myminio/images
mc mb mysilo/images
mc event add mysilo/images arn:minio:sqs::1:kafka --suffix .jpg
[Webhooks](https://en.wikipedia.org/wiki/Webhook) are a way to receive information when it happens, rather than continually polling for that data.
### Step 1: Add Webhook endpoint to MinIO
### Step 1: Add Webhook endpoint to Silo
MinIO supports persistent event store. The persistent store will backup events when the webhook goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the webhook goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
```
KEY:
@@ -1291,30 +1296,30 @@ MINIO_NOTIFY_WEBHOOK_CLIENT_KEY (string) client cert key for Webhook mTLS a
Use `mc admin config set` command to update the configuration for the deployment. Here the endpoint is the server listening for webhook notifications. Save the settings and restart the MinIO server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your MinIO server.
Use `mc admin config set` command to update the configuration for the deployment. Here the endpoint is the server listening for webhook notifications. Save the settings and restart the Silo server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your Silo server.
```sh
mc admin config set myminio notify_webhook:1 queue_limit="0"endpoint="http://localhost:3000"queue_dir=""
mc admin config set mysilo notify_webhook:1 queue_limit="0"endpoint="http://localhost:3000"queue_dir=""
```
### Step 2: Enable Webhook bucket notification using MinIO client
### Step 2: Enable Webhook bucket notification using Silo client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded to `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:webhook`. To learn more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded to `images` bucket on `mysilo` server. Here ARN value is `arn:minio:sqs::1:webhook`. To learn more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
mc mb myminio/images-thumbnail
mc event add myminio/images arn:minio:sqs::1:webhook --event put --suffix .jpg
mc mb mysilo/images
mc mb mysilo/images-thumbnail
mc event add mysilo/images arn:minio:sqs::1:webhook --event put --suffix .jpg
```
Check if event notification is successfully configured by
We used [Thumbnailer](https://github.com/minio/thumbnailer) to listen for MinIO notifications when a new JPEG file is uploaded (HTTP PUT). Triggered by a notification, Thumbnailer uploads a thumbnail of new image to MinIO server. To start with, download and install Thumbnailer.
We used [Thumbnailer](https://github.com/minio/thumbnailer) to listen for Silo notifications when a new JPEG file is uploaded (HTTP PUT). Triggered by a notification, Thumbnailer uploads a thumbnail of new image to Silo server. To start with, download and install Thumbnailer.
```
git clone https://github.com/minio/thumbnailer/
npm install
```
Then open the Thumbnailer config file at `config/webhook.json` and add the configuration for your MinIO server and then start Thumbnailer by
Then open the Thumbnailer config file at `config/webhook.json` and add the configuration for your Silo server and then start Thumbnailer by
```
NODE_ENV=webhook node thumbnail-webhook.js
```
Thumbnailer starts running at `http://localhost:3000/`. Next, configure the MinIO server to send notifications to this URL (as mentioned in step 1) and use `mc` to set up bucket notifications (as mentioned in step 2). Then upload a JPEG image to MinIO server by
Thumbnailer starts running at `http://localhost:3000/`. Next, configure the Silo server to send notifications to this URL (as mentioned in step 1) and use `mc` to set up bucket notifications (as mentioned in step 2). Then upload a JPEG image to Silo server by
Install an NSQ Daemon from [here](https://nsq.io/). Or use the following Docker
command for starting an nsq daemon:
@@ -1361,9 +1366,9 @@ command for starting an nsq daemon:
podman run --rm -p 4150-4151:4150-4151 nsqio/nsq /nsqd
```
### Step 1: Add NSQ endpoint to MinIO
### Step 1: Add NSQ endpoint to Silo
MinIO supports persistent event store. The persistent store will backup events when the NSQ broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
Silo supports persistent event store. The persistent store will backup events when the NSQ broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 100000.
To update the configuration, use `mc admin config get` command to get the current configuration for `notify_nsq`.
@@ -1399,26 +1404,26 @@ MINIO_NOTIFY_NSQ_COMMENT (sentence) optionally add a comment to this s
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the Silo server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
```sh
mc admin config set myminio notify_nsq:1 nsqd_address="127.0.0.1:4150"queue_dir=""queue_limit="0"tls="off"tls_skip_verify="on"topic="minio"
mc admin config set mysilo notify_nsq:1 nsqd_address="127.0.0.1:4150"queue_dir=""queue_limit="0"tls="off"tls_skip_verify="on"topic="minio"
```
Note that, you can add as many NSQ daemon endpoint configurations as needed by providing an identifier (like "1" in the example above) for the NSQ instance and an object of per-server configuration parameters.
### Step 2: Enable NSQ bucket notification using MinIO client
### Step 2: Enable NSQ bucket notification using Silo client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nsq`.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `mysilo` server. Here ARN value is `arn:minio:sqs::1:nsq`.
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:nsq --suffix .jpg
mc event list myminio/images
mc mb mysilo/images
mc event add mysilo/images arn:minio:sqs::1:nsq --suffix .jpg
This document explains the design approach of server side bucket replication. If you're looking to get started with replication, we suggest you go through the [Bucket replication guide](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/README.md) first.
@@ -12,13 +12,13 @@ If an object meets replication rules as set in the replication configuration, `X
All replication failures are picked up by the scanner which runs at a one minute frequency, each time scanning up to a sixteenth of the namespace. Object versions marked `PENDING` or `FAILED` are re-queued for replication.
Replication speed depends on the cluster load, number of objects in the object store as well as storage speed. In addition, any bandwidth limits set via `mc admin bucket remote add` could also contribute to replication speed. The number of workers used for replication defaults to 100. Based on network bandwidth and system load, the number of workers used in replication can be configured using `mc admin config set alias api` to set the `replication_workers`. The prometheus metrics exposed by MinIO can be used to plan resource allocation and bandwidth management to optimize replication speed.
Replication speed depends on the cluster load, number of objects in the object store as well as storage speed. In addition, any bandwidth limits set via `mc admin bucket remote add` could also contribute to replication speed. The number of workers used for replication defaults to 100. Based on network bandwidth and system load, the number of workers used in replication can be configured using `mc admin config set alias api` to set the `replication_workers`. The prometheus metrics exposed by Silo can be used to plan resource allocation and bandwidth management to optimize replication speed.
If synchronous replication is configured above, replication is attempted right away prior to returning the PUT object response. In the event that the replication target is down, the `X-Amz-Replication-Status` is marked as `FAILED` and resynced with target when the scanner runs again.
Any metadata changes on the source object version, such as metadata updates via PutObjectTagging, PutObjectRetention, PutObjectLegalHold and COPY api are replicated in a similar manner to target version, with the `X-Amz-Replication-Status` again cycling through the same states.
The description above details one way replication from source to target w.r.t incoming object uploads and metadata changes to source object version. If active-active replication is configured, any incoming uploads and metadata changes to versions created on the target, will sync back to the source and be marked as `REPLICA` on the source. AWS, as well as MinIO do not by default sync metadata changes on a object version marked `REPLICA` back to source. This requires a setting in the replication configuration called [replica modification sync](https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-replication-adds-support-two-way-replication/).
The description above details one way replication from source to target w.r.t incoming object uploads and metadata changes to source object version. If active-active replication is configured, any incoming uploads and metadata changes to versions created on the target, will sync back to the source and be marked as `REPLICA` on the source. AWS, as well as Silo do not by default sync metadata changes on a object version marked `REPLICA` back to source. This requires a setting in the replication configuration called [replica modification sync](https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-replication-adds-support-two-way-replication/).
For active-active replication, automatic failover occurs on `GET/HEAD` operations if object or object version requested qualifies for replication and is missing on one site, but present on the other. This allows the applications to take full advantage of two-way replication even before the two sites get fully synced.
@@ -26,7 +26,7 @@ In the case of multi destination replication, the replication status shows `COMP
### Replication of DeleteMarker and versioned Delete
MinIO allows DeleteMarker replication and versioned delete replication by setting `--replicate delete,delete-marker` while setting up replication configuration using `mc replicate add`. The MinIO implementation is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
Silo allows DeleteMarker replication and versioned delete replication by setting `--replicate delete,delete-marker` while setting up replication configuration using `mc replicate add`. The Silo implementation is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
Similar to object version replication, DeleteMarker replication also cycles through `PENDING` to `COMPLETED` or `FAILED` states for the `X-Amz-Replication-Status` on the source when a delete marker is set (i.e. performing `mc rm` on an object without specifying a version).After replication syncs the delete marker on the target, the DeleteMarker on the target shows `X-Amz-Replication-Status` of `REPLICA`. The status of DeleteMarker replication is returned by `X-Minio-Replication-DeleteMarker-Status` header on `HEAD/GET` calls for the delete marker version in question - i.e with `mc stat --version-id dm-version-id`
@@ -156,5 +156,5 @@ If 3 or more targets are participating in active-active replication, the replica
Bucket replication is designed to replicate selected objects in a bucket to a destination bucket.
The contents of this page have been migrated to the new [MinIO Documentation: Bucket Replication](https://silo.pgsty.com/administration/bucket-replication/) page. The [Bucket Replication](https://silo.pgsty.com/administration/bucket-replication/bucket-replication-requirements/) page references dedicated tutorials for configuring one-way "Active-Passive" and two-way "Active-Active" bucket replication.
The contents of this page have been migrated to the new [Silo Documentation: Bucket Replication](https://silo.pgsty.com/administration/bucket-replication/) page. The [Bucket Replication](https://silo.pgsty.com/administration/bucket-replication/bucket-replication-requirements/) page references dedicated tutorials for configuring one-way "Active-Passive" and two-way "Active-Active" bucket replication.
To replicate objects in a bucket to a destination bucket on a target site either in the same cluster or a different cluster, start by enabling [versioning](https://silo.pgsty.com/administration/object-management/object-versioning/) for both source and destination buckets. Finally, the target site and the destination bucket need to be configured on the source MinIO server.
To replicate objects in a bucket to a destination bucket on a target site either in the same cluster or a different cluster, start by enabling [versioning](https://silo.pgsty.com/administration/object-management/object-versioning/) for both source and destination buckets. Finally, the target site and the destination bucket need to be configured on the source Silo server.
## Highlights
- Supports source and destination buckets to have the same name unlike AWS S3, addresses variety of use-cases such as *Splunk*, *Veeam* site to site DR.
- Supports object locking/retention across source and destination buckets natively out of the box, unlike AWS S3.
- Simpler implementation than [AWS S3 Bucket Replication Config](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html) with requirements such as IAM Role, AccessControlTranslation, Metrics and SourceSelectionCriteria are not needed with MinIO.
- Simpler implementation than [AWS S3 Bucket Replication Config](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html) with requirements such as IAM Role, AccessControlTranslation, Metrics and SourceSelectionCriteria are not needed with Silo.
- Active-Active replication
- Multi destination replication
@@ -96,21 +96,22 @@ The access key provided for the replication *target* cluster should have these m
Please note that the permissions required by the admin user on the target cluster can be more fine grained to exclude permissions like "s3:ReplicateDelete", "s3:GetBucketObjectLockConfiguration" etc depending on whether delete replication rules are set up or if object locking is disabled on `destbucket`. The above policies assume that replication of objects, tags and delete marker replication are all enabled on object lock enabled buckets. A sample script to setup replication is provided [here](https://github.com/pgsty/minio/blob/master/docs/bucket/replication/setup_replication.sh)
To set up replication from a source bucket `srcbucket` on myminio cluster to a bucket `destbucket` on the target minio cluster with endpoint https://replica-endpoint:9000, use:
To set up replication from `srcbucket` on the `mysilo` cluster to `destbucket`
on a target Silo cluster at `https://replica-endpoint:9000`, use:
```
mc replicate add myminio/srcbucket --priority 1 --remote-bucket https://accessKey:secretKey@replica-endpoint:9000/destbucket
Replication configuration applied successfully to myminio/srcbucket.
mc replicate add mysilo/srcbucket --priority 1 --remote-bucket https://accessKey:secretKey@replica-endpoint:9000/destbucket
Replication configuration applied successfully to mysilo/srcbucket.
```
Internally, this creates an ARN for the remote target associating the remote bucket as a replication target to the srcbucket on myminio.By default, if --replicate flag is not specified, replication of delete marker, permanent deletes, existing object replication and replica modification sync are all enabled. If you are using older mc versions, the ARN needs to be generated as a separate step before adding a replication rule.
Internally, this creates an ARN for the remote target associating the remote bucket as a replication target to the srcbucket on mysilo.By default, if --replicate flag is not specified, replication of delete marker, permanent deletes, existing object replication and replica modification sync are all enabled. If you are using older mc versions, the ARN needs to be generated as a separate step before adding a replication rule.
> NOTE: If you are using a mc version below `RELEASE.2022-12-24T15-21-38Z`, the --remote-bucket flag needs an ARN generated by `mc admin bucket remote add` command. For mc versions RELEASE.2021-09-02T09-21-27Z and older, the remote target ARN needs to be passed in the --arn flag and actual remote bucket name in --remote-bucket flag of `mc replicate add`. For example, in older releases of mc replication configuration used to be added with:
Replication configuration applied successfully to mysilo/srcbucket.
```
The replication configuration generated has the following format and can be exported with `mc replicate export` command:
@@ -153,7 +154,7 @@ The replication configuration generated has the following format and can be expo
}
```
The replication configuration follows [AWS S3 Spec](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html). Any objects uploaded to the source bucket that meet replication criteria will now be automatically replicated by the MinIO server to the remote destination bucket. Replication can be disabled at any time by disabling specific rules in the configuration or deleting the replication configuration entirely.
The replication configuration follows [AWS S3 Spec](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html). Any objects uploaded to the source bucket that meet replication criteria will now be automatically replicated by the Silo server to the remote destination bucket. Replication can be disabled at any time by disabling specific rules in the configuration or deleting the replication configuration entirely.
When object locking is used in conjunction with replication, both source and destination buckets needs to have [object locking](https://silo.pgsty.com/administration/object-management/object-retention/) enabled. Similarly objects encrypted on the server side, will be replicated if destination also supports encryption.
@@ -167,7 +168,7 @@ To perform bi-directional replication, repeat the above process on the target si
## Replica Modification sync
If bi-directional replication is set up between two clusters, any metadata update on the REPLICA object is by default reflected back in the source object when `ReplicaModifications` status in the `SourceSelectionCriteria` is `Enabled`. In MinIO, this is enabled by default. If a metadata update is performed on the "REPLICA" object, its `X-Amz-Replication-Status` will change from `PENDING` to `COMPLETE` or `FAILED`, and the source object version will show `X-Amz-Replication-Status` of `REPLICA` once the replication operation is complete.
If bi-directional replication is set up between two clusters, any metadata update on the REPLICA object is by default reflected back in the source object when `ReplicaModifications` status in the `SourceSelectionCriteria` is `Enabled`. In Silo, this is enabled by default. If a metadata update is performed on the "REPLICA" object, its `X-Amz-Replication-Status` will change from `PENDING` to `COMPLETE` or `FAILED`, and the source object version will show `X-Amz-Replication-Status` of `REPLICA` once the replication operation is complete.
The replication configuration in use on a bucket can be viewed using the `mc replicate export alias/bucket` command.
@@ -183,25 +184,25 @@ To re-enable replica metadata modification syncing,
mc replicate edit alias/bucket --id xyz.id --replicate "delete,delete-marker,replica-metadata-sync"
```
## MinIO Extension
## Silo Extension
### Replicating Deletes
Delete marker replication is allowed in [AWS V1 Configuration](https://aws.amazon.com/blogs/storage/managing-delete-marker-replication-in-amazon-s3/) but not in V2 configuration. The MinIO implementation above is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration above. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
Delete marker replication is allowed in [AWS V1 Configuration](https://aws.amazon.com/blogs/storage/managing-delete-marker-replication-in-amazon-s3/) but not in V2 configuration. The Silo implementation above is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration above. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
When an object is deleted from the source bucket, the corresponding replica version will be marked deleted if delete marker replication is enabled in the replication configuration. Replication of deletes that specify a version id (a.k.a hard deletes) can be enabled by setting the `DeleteReplication` status to enabled in the replication configuration. This is a MinIO specific extension that can be enabled using the `mc replicate add` or `mc replicate edit` command with the --replicate "delete" flag.
When an object is deleted from the source bucket, the corresponding replica version will be marked deleted if delete marker replication is enabled in the replication configuration. Replication of deletes that specify a version id (a.k.a hard deletes) can be enabled by setting the `DeleteReplication` status to enabled in the replication configuration. This is a Silo specific extension that can be enabled using the `mc replicate add` or `mc replicate edit` command with the --replicate "delete" flag.
Note that due to this extension behavior, AWS SDK's may not support the extension functionality pertaining to replicating versioned deletes.
Note that just like with [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-marker-replication.html), Delete marker replication is disallowed in MinIO when the replication rule has tags.
Note that just like with [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-marker-replication.html), Delete marker replication is disallowed in Silo when the replication rule has tags.
To add a replication rule allowing both delete marker replication, versioned delete replication or both specify the --replicate flag with comma separated values as in the example below.
Additional permission of "s3:ReplicateDelete" action would need to be specified on the access key configured for the target cluster if Delete Marker replication or versioned delete replication is enabled.
Replication configuration applied successfully to mysilo/srcbucket.
```
> NOTE: In mc versions `RELEASE.2022-12-24T15-21-38Z` and above `remote-target` should be of the format `https://accessKey:secretKey@replica-endpoint:9000/destbucket` which earlier used to be set during `mc admin bucket remote add`. For older releases, use the arn generated with `mc admin bucket remote add` command - e.g."arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket" as the `remote-target`.
@@ -223,7 +224,7 @@ remote replication target using the `mc admin bucket remote add` command. For mc
--sync, --health-check and --bandwidth flags can be specified in `mc replicate add|update` command
@@ -266,16 +267,16 @@ In the above sample config, objects under prefixes matching any of the `Excluded
### SSE-C Encryption
MinIO does not support SSE-C encrypted objects on replicated buckets, any application uploading SSE-C encrypted objects will be rejected with an error on replicated buckets.
Silo does not support SSE-C encrypted objects on replicated buckets, any application uploading SSE-C encrypted objects will be rejected with an error on replicated buckets.
#### Rationale
- SSE-C requires application to remember the keys for all GET/PUT operations, any unfortunate loss of keys would automatically mean the objects cannot be accessed anymore.
- SSE-C is hardly adopted by most widely used applications, applications prefer server to manage the keys via SSE-KMS or SSE-S3.
-MinIO recommends applications to use SSE-KMS, SSE-S3 for simpler, safer and robust encryption mechanism for replicated buckets.
-Silo recommends applications to use SSE-KMS, SSE-S3 for simpler, safer and robust encryption mechanism for replicated buckets.
# Object Lock and Immutability Guide [](https://slack.min.io)
# Object Lock and Immutability Guide
MinIO server allows WORM for specific objects or by configuring a bucket with default object lock configuration that applies default retention mode and retention duration to all objects. This makes objects in the bucket immutable i.e. delete of the version are not allowed until an expiry specified in the bucket's object lock configuration or object retention.
Silo server allows WORM for specific objects or by configuring a bucket with default object lock configuration that applies default retention mode and retention duration to all objects. This makes objects in the bucket immutable i.e. delete of the version are not allowed until an expiry specified in the bucket's object lock configuration or object retention.
Object locking requires locking to be enabled on a bucket at the time of bucket creation refer to `mc mb --with-lock`, object locking enables versioning on the bucket and cannot be disabled.
@@ -10,7 +10,7 @@ A default retention period and retention mode can be configured on a bucket to b
MinIO versioning is designed to keep multiple versions of an object in one bucket. For example, you could store `spark.csv` (version `ede336f2`) and `spark.csv` (version `fae684da`) in a single bucket. Versioning protects you from unintended overwrites, deletions, protect objects with retention policies.
Silo versioning is designed to keep multiple versions of an object in one bucket. For example, you could store `spark.csv` (version `ede336f2`) and `spark.csv` (version `fae684da`) in a single bucket. Versioning protects you from unintended overwrites, deletions, protect objects with retention policies.
To control data retention and storage usage, use object versioning with [object lifecycle management](https://github.com/pgsty/minio/blob/master/docs/bucket/lifecycle/README.md). If you have an object expiration lifecycle policy in your non-versioned bucket and you want to maintain the same permanent delete behavior when on versioning-enabled bucket, you must add a noncurrent expiration policy. The noncurrent expiration lifecycle policy will manage the deletes of the noncurrent object versions in the versioning-enabled bucket. (A version-enabled bucket maintains one current and zero or more noncurrent object versions.)
Versioning must be explicitly enabled on a bucket, versioning is not enabled by default. Object locking enabled buckets have versioning enabled automatically. Enabling and suspending versioning is done at the bucket level.
Only MinIO generates version IDs, and they can't be edited. Version IDs are simply of `DCE 1.1 v4 UUID 4` (random data based), UUIDs are 128 bit numbers which are intended to have a high likelihood of uniqueness over space and time and are computationally difficult to guess. They are globally unique identifiers which can be locally generated without contacting a global registration authority. UUIDs are intended as unique identifiers for both mass tagging objects with an extremely short lifetime and to reliably identifying very persistent objects across a network.
Only Silo generates version IDs, and they can't be edited. Version IDs are simply of `DCE 1.1 v4 UUID 4` (random data based), UUIDs are 128 bit numbers which are intended to have a high likelihood of uniqueness over space and time and are computationally difficult to guess. They are globally unique identifiers which can be locally generated without contacting a global registration authority. UUIDs are intended as unique identifiers for both mass tagging objects with an extremely short lifetime and to reliably identifying very persistent objects across a network.
When you PUT an object in a versioning-enabled bucket, the noncurrent version is not overwritten. The following figure shows that when a new version of `spark.csv` is PUT into a bucket that already contains an object with the same name, the original object (ID = `ede336f2`) remains in the bucket, MinIO generates a new version (ID = `fae684da`), and adds the newer version to the bucket.
When you PUT an object in a versioning-enabled bucket, the noncurrent version is not overwritten. The following figure shows that when a new version of `spark.csv` is PUT into a bucket that already contains an object with the same name, the original object (ID = `ede336f2`) remains in the bucket, Silo generates a new version (ID = `fae684da`), and adds the newer version to the bucket.
@@ -32,7 +32,7 @@ To permanently delete an object you need to specify the version you want to dele
## Concepts
- All Buckets on MinIO are always in one of the following states: unversioned (the default) and all other existing deployments, versioning-enabled, or versioning-suspended.
- All Buckets on Silo are always in one of the following states: unversioned (the default) and all other existing deployments, versioning-enabled, or versioning-suspended.
- Versioning state applies to all of the objects in the versioning enabled bucket. The first time you enable a bucket for versioning, objects in the bucket are thereafter always versioned and given a unique version ID.
- Existing or newer buckets can be created with versioning enabled and eventually can be suspended as well. Existing versions of objects stay as is and can still be accessed using the version ID.
- All versions, including delete-markers should be deleted before deleting a bucket.
@@ -47,7 +47,7 @@ Each bucket created has a versioning configuration associated with it. By defaul
</VersioningConfiguration>
```
To enable versioning, you send a request to MinIO with a versioning configuration with Status set to `Enabled`.
To enable versioning, you send a request to Silo with a versioning configuration with Status set to `Enabled`.
@@ -63,7 +63,7 @@ Similarly to suspend versioning set the configuration with Status set to `Suspen
</VersioningConfiguration>
```
## MinIO extension to Bucket Versioning
## Silo extension to Bucket Versioning
### Idempotent versions on directory objects
@@ -73,13 +73,15 @@ All directory objects such as objects that end with `/`, will only have one vers
### Idempotent versions on delete markers
Duplicate delete markers are not created on MinIO buckets with versioning, if an application performs a soft delete on an object repeatedly - that object will only ever have a single DELETE marker for all such successive attempts. This is done to ensure that repeated soft deletes do not ever need multiple versions in the first place.
Duplicate delete markers are not created on Silo buckets with versioning, if an application performs a soft delete on an object repeatedly - that object will only ever have a single DELETE marker for all such successive attempts. This is done to ensure that repeated soft deletes do not ever need multiple versions in the first place.
> NOTE: Server side replication is supported for idempotent versions on delete marked objects.
### Motivation
**PLEASE READ: This feature is meant for advanced use-cases only where the setup is using bucket versioning or with replicated buckets, use this feature to optimize versioning behavior for some specific applications. MinIO experts will evaluate and guide on the benefits for your application, please reach out to us on <https://subnet.min.io>.**
**PLEASE READ:** This advanced behavior is intended for carefully evaluated
versioned or replicated workloads. Validate its lifecycle and replication
effects in a staging environment before enabling it in production.
Spark/Hadoop workloads which use Hadoop MR Committer v1/v2 algorithm upload objects to a temporary prefix in a bucket. These objects are 'renamed' to a different prefix on Job commit. Object storage admins are forced to configure separate ILM policies to expire these objects and their versions to reclaim space.
@@ -110,7 +112,7 @@ To exclude objects under a list of prefix (glob) patterns from being versioned,
- Objects matching these prefixes will also not leave `null` delete markers, dramatically reduces namespace pollution while keeping the benefits of replication.
- Users with explicit permissions or the root credential can configure the versioning state of any bucket.
## Examples of enabling bucket versioning using MinIO Java SDK
## Examples using the MinIO Java SDK compatibility client
### EnableVersioning() API
@@ -127,11 +129,11 @@ public class EnableVersioning {
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Feng Ruohang