minio/docs/batch-jobs/README.md

# Silo Batch Job
Silo Batch jobs is an Silo object management feature that lets you manage objects at scale. Jobs currently supported by Silo

- Replicate objects between buckets on multiple sites

Upcoming Jobs

- Copy objects from NAS to Silo
- Copy objects from HDFS to Silo

## Replication Job
To perform replication via batch jobs, you create a job. The job consists of a job description YAML that describes

- Source location from where the objects must be copied from
- Target location from where the objects must be copied to
- Fine grained filtering is available to pick relevant objects from source to copy from

Silo batch jobs framework also provides

- Retrying a failed job automatically driven by user input
- Monitoring job progress in real-time
- Send notifications upon completion or failure to user configured target

Following YAML describes the structure of a replication job, each value is documented and self-describing.

```yaml
replicate:
  apiVersion: v1
  # source of the objects to be replicated
  source:
	type: TYPE # valid values are "minio"
	bucket: BUCKET
	prefix: PREFIX
	# NOTE: if source is remote then target must be "local"
	# endpoint: ENDPOINT
	# credentials:
	#   accessKey: ACCESS-KEY
	#   secretKey: SECRET-KEY
	#   sessionToken: SESSION-TOKEN # Available when rotating credentials are used

  # target where the objects must be replicated
  target:
	type: TYPE # valid values are "minio"
	bucket: BUCKET
	prefix: PREFIX
	# NOTE: if target is remote then source must be "local"
	# endpoint: ENDPOINT
	# credentials:
	#   accessKey: ACCESS-KEY
	#   secretKey: SECRET-KEY
	#   sessionToken: SESSION-TOKEN # Available when rotating credentials are used

  # optional flags based filtering criteria
  # for all source objects
  flags:
	filter:
	  newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
	  olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
	  createdAfter: "date" # match objects created after "date"
	  createdBefore: "date" # match objects created before "date"

	  ## NOTE: tags are not supported when "source" is remote.
	  # tags:
	  #   - key: "name"
	  #     value: "pick*" # match objects with tag 'name', with all values starting with 'pick'

	  ## NOTE: metadata filter not supported when "source" is non Silo.
	  # metadata:
	  #   - key: "content-type"
	  #     value: "image/*" # match objects with 'content-type', with all values starting with 'image/'

	notify:
	  endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
	  token: "Bearer xxxxx" # optional authentication token for the notification endpoint

	retry:
	  attempts: 10 # number of retries for the job before giving up
	  delay: "500ms" # least amount of delay between each retry
```

You can create and run multiple 'replication' jobs at a time there are no predefined limits set.

## Batch Jobs Terminology

### Job
A job is the basic unit of work for Silo Batch Job. A job is a self describing YAML, once this YAML is submitted and evaluated - Silo performs the requested actions on each of the objects obtained under the described criteria in job YAML file.

### Type
Type describes the job type, such as replicating objects between Silo sites. Each job performs a single type of operation across all objects that match the job description criteria.

## Batch Jobs via Commandline
The compatible [`mc`](https://github.com/pgsty/mc) client provides the
`mc batch` command to create, start, and manage submitted jobs.

```
NAME:
  mc batch - manage batch jobs

USAGE:
  mc batch COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]

COMMANDS:
  generate  generate a new batch job definition
  start     start a new batch job
  list, ls  list all current batch jobs
  status    summarize job events on Silo server in real-time
  describe  describe job definition for a job
```

### Generate a job yaml
```
mc batch generate alias/ replicate
```

### Start the batch job (returns back the JID)
```
mc batch start alias/ ./replicate.yaml
Successfully start 'replicate' job `E24HH4nNMcgY5taynaPfxu` on '2022-09-26 17:19:06.296974771 -0700 PDT'
```

### List all batch jobs
```
mc batch list alias/
ID                      TYPE            USER            STARTED
E24HH4nNMcgY5taynaPfxu  replicate       minioadmin      1 minute ago
```

### List all 'replicate' batch jobs
```
mc batch list alias/ --type replicate
ID                      TYPE            USER            STARTED
E24HH4nNMcgY5taynaPfxu  replicate       minioadmin      1 minute ago
```

### Real-time 'status' for a batch job
```
mc batch status mysilo/ E24HH4nNMcgY5taynaPfxu
●∙∙
Objects:        28766
Versions:       28766
Throughput:     3.0 MiB/s
Transferred:    406 MiB
Elapsed:        2m14.227222868s
CurrObjName:    share/doc/xml-core/examples/foo.xmlcatalogs
```

### 'describe' the batch job yaml.
```
mc batch describe mysilo/ E24HH4nNMcgY5taynaPfxu
replicate:
  apiVersion: v1
...
```