mirror of
https://github.com/pgsty/minio.git
synced 2026-08-09 15:53:28 +03:00
docs: rebrand the repository documentation, templates and dashboards
README, README_ZH, SECURITY, COMPLIANCE, CONTRIBUTING, NOTICE, code_of_conduct, the vulnerability and PR-etiquette documents, the GitHub issue and pull request templates, and the docs/ tree all present Silo as the product. The Grafana dashboards under docs/metrics/prometheus/grafana/ have their panel titles and descriptions rebranded while every minio_* query, label and expression is left alone, so existing alerts and recording rules keep matching. The distinction the review demanded is applied per hit rather than by search-and-replace: - Product and command text becomes Silo and silo: install and run instructions, systemd examples, compose services, download links, badges. - Protocol and interface text keeps MinIO: MINIO_* variables, minio_* metrics, x-minio-* headers, /minio/* routes, .minio.sys, arn:minio, and API field and error names. - Attribution keeps MinIO and gains the fork's own: the AGPL obligations, original copyright, CREDITS and NOTICE stay, with the modification notice added alongside rather than replacing them. - Historical and third-party references are left as facts, not rewritten for brand tidiness. README and README_ZH each carry an explicit non-affiliation notice, document the side-by-side package migration including the /etc/systemd/system/silo.service.d/10-legacy-user.conf drop-in for keeping a legacy UID/GID, and state that recursive chown is never performed. The trademark attribution uses the policy's approved "based on MinIO technology" wording, not the shortened form the policy rejects. github.com/pgsty/minio links are left in place and labelled transitional. The repository has not been renamed, and rewriting them now would produce documented URLs that 404 until the cutover; they change in the cutover commit together with the goreleaser release target, the OCI source label and the raw-content branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Feng Ruohang
+20
-20
@@ -1,4 +1,4 @@
|
||||
# **Disaggregated HDP Spark and Hive with MinIO**
|
||||
# **Disaggregated HDP Spark and Hive with Silo**
|
||||
|
||||
## **1. Cloud-native Architecture**
|
||||
|
||||
@@ -6,20 +6,20 @@
|
||||
|
||||
Kubernetes manages stateless Spark and Hive containers elastically on the compute nodes. Spark has native scheduler integration with Kubernetes. Hive, for legacy reasons, uses YARN scheduler on top of Kubernetes.
|
||||
|
||||
All access to MinIO object storage is via S3/SQL SELECT API. In addition to the compute nodes, MinIO containers are also managed by Kubernetes as stateful containers with local storage (JBOD/JBOF) mapped as persistent local volumes. This architecture enables multi-tenant MinIO, allowing isolation of data between customers.
|
||||
All access to Silo object storage is via S3/SQL SELECT API. In addition to the compute nodes, Silo containers are also managed by Kubernetes as stateful containers with local storage (JBOD/JBOF) mapped as persistent local volumes. This architecture enables multi-tenant Silo, allowing isolation of data between customers.
|
||||
|
||||
MinIO also supports multi-cluster, multi-site federation similar to AWS regions and tiers. Using MinIO Information Lifecycle Management (ILM), you can configure data to be tiered between NVMe based hot storage, and HDD based warm storage. All data is encrypted with per-object key. Access Control and Identity Management between the tenants are managed by MinIO using OpenID Connect or Kerberos/LDAP/AD.
|
||||
Silo also supports multi-cluster, multi-site federation similar to AWS regions and tiers. Using Silo Information Lifecycle Management (ILM), you can configure data to be tiered between NVMe based hot storage, and HDD based warm storage. All data is encrypted with per-object key. Access Control and Identity Management between the tenants are managed by Silo using OpenID Connect or Kerberos/LDAP/AD.
|
||||
|
||||
## **2. Prerequisites**
|
||||
|
||||
- Install Hortonworks Distribution using this [guide.](https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/ch_Installing_Ambari.html)
|
||||
- [Setup Ambari](https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/set_up_the_ambari_server.html) which automatically sets up YARN
|
||||
- [Installing Spark](https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/installing-spark/content/installing_spark.html)
|
||||
- Install MinIO Distributed Server using one of the guides below.
|
||||
- Install Silo Distributed Server using one of the guides below.
|
||||
- [Deployment based on Kubernetes](https://silo.pgsty.com/operations/deployments/kubernetes/)
|
||||
- [Deployment based on MinIO Helm Chart](https://github.com/helm/charts/tree/master/stable/minio)
|
||||
- [Deployment based on the Silo Helm chart](../../helm/silo/README.md)
|
||||
|
||||
## **3. Configure Hadoop, Spark, Hive to use MinIO**
|
||||
## **3. Configure Hadoop, Spark, Hive to use Silo**
|
||||
|
||||
After successful installation navigate to the Ambari UI `http://<ambari-server>:8080/` and login using the default credentials: [**_username: admin, password: admin_**]
|
||||
|
||||
@@ -31,7 +31,7 @@ Navigate to **Services** -> **HDFS** -> **CONFIGS** -> **ADVANCED** as shown bel
|
||||
|
||||
|
||||
|
||||
Navigate to **Custom core-site** to configure MinIO parameters for `_s3a_` connector
|
||||
Navigate to **Custom core-site** to configure Silo parameters for `_s3a_` connector
|
||||
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ sudo pip install yq
|
||||
alias kv-pairify='yq ".configuration[]" | jq ".[]" | jq -r ".name + \"=\" + .value"'
|
||||
```
|
||||
|
||||
Let's take for example a set of 12 compute nodes with an aggregate memory of _1.2TiB_, we need to do following settings for optimal results. Add the following optimal entries for _core-site.xml_ to configure _s3a_ with **MinIO**. Most important options here are
|
||||
Let's take for example a set of 12 compute nodes with an aggregate memory of _1.2TiB_, we need to do following settings for optimal results. Add the following optimal entries for _core-site.xml_ to configure _s3a_ with **Silo**. Most important options here are
|
||||
|
||||
```
|
||||
cat ${HADOOP_CONF_DIR}/core-site.xml | kv-pairify | grep "mapred"
|
||||
@@ -56,7 +56,7 @@ mapreduce.task.io.sort.factor=999 # Threshold before writing to disk
|
||||
mapreduce.task.sort.spill.percent=0.9 # Minimum % before spilling to disk
|
||||
```
|
||||
|
||||
S3A is the connector to use S3 and other S3-compatible object stores such as MinIO. MapReduce workloads typically interact with object stores in the same way they do with HDFS. These workloads rely on HDFS atomic rename functionality to complete writing data to the datastore. Object storage operations are atomic by nature and they do not require/implement rename API. The default S3A committer emulates renames through copy and delete APIs. This interaction pattern causes significant loss of performance because of the write amplification. _Netflix_, for example, developed two new staging committers - the Directory staging committer and the Partitioned staging committer - to take full advantage of native object storage operations. These committers do not require rename operation. The two staging committers were evaluated, along with another new addition called the Magic committer for benchmarking.
|
||||
S3A is the connector to use S3 and other S3-compatible object stores such as Silo. MapReduce workloads typically interact with object stores in the same way they do with HDFS. These workloads rely on HDFS atomic rename functionality to complete writing data to the datastore. Object storage operations are atomic by nature and they do not require/implement rename API. The default S3A committer emulates renames through copy and delete APIs. This interaction pattern causes significant loss of performance because of the write amplification. _Netflix_, for example, developed two new staging committers - the Directory staging committer and the Partitioned staging committer - to take full advantage of native object storage operations. These committers do not require rename operation. The two staging committers were evaluated, along with another new addition called the Magic committer for benchmarking.
|
||||
|
||||
It was found that the directory staging committer was the fastest among the three, S3A connector should be configured with the following parameters for optimal results:
|
||||
|
||||
@@ -64,7 +64,7 @@ It was found that the directory staging committer was the fastest among the thre
|
||||
cat ${HADOOP_CONF_DIR}/core-site.xml | kv-pairify | grep "s3a"
|
||||
|
||||
fs.s3a.access.key=minio
|
||||
fs.s3a.secret.key=minio123
|
||||
fs.s3a.secret.key=silo123
|
||||
fs.s3a.path.style.access=true
|
||||
fs.s3a.block.size=512M
|
||||
fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a
|
||||
@@ -80,7 +80,7 @@ fs.s3a.connection.timeout=200000
|
||||
fs.s3a.endpoint=http://minio:9000
|
||||
fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
|
||||
|
||||
fs.s3a.committer.threads=2048 # Number of threads writing to MinIO
|
||||
fs.s3a.committer.threads=2048 # Number of threads writing to Silo
|
||||
fs.s3a.connection.maximum=8192 # Maximum number of concurrent conns
|
||||
fs.s3a.fast.upload.active.blocks=2048 # Number of parallel uploads
|
||||
fs.s3a.fast.upload.buffer=disk # Use disk as the buffer for uploads
|
||||
@@ -108,15 +108,15 @@ Navigate to **Services** -> **Spark2** -> **CONFIGS** as shown below
|
||||
|
||||
|
||||
|
||||
Navigate to “**Custom spark-defaults**” to configure MinIO parameters for `_s3a_` connector
|
||||
Navigate to “**Custom spark-defaults**” to configure Silo parameters for `_s3a_` connector
|
||||
|
||||
|
||||
|
||||
Add the following optimal entries for _spark-defaults.conf_ to configure Spark with **MinIO**.
|
||||
Add the following optimal entries for _spark-defaults.conf_ to configure Spark with **Silo**.
|
||||
|
||||
```
|
||||
spark.hadoop.fs.s3a.access.key minio
|
||||
spark.hadoop.fs.s3a.secret.key minio123
|
||||
spark.hadoop.fs.s3a.secret.key silo123
|
||||
spark.hadoop.fs.s3a.path.style.access true
|
||||
spark.hadoop.fs.s3a.block.size 512M
|
||||
spark.hadoop.fs.s3a.buffer.dir ${hadoop.tmp.dir}/s3a
|
||||
@@ -126,7 +126,7 @@ spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads true
|
||||
spark.hadoop.fs.s3a.committer.staging.conflict-mode append
|
||||
spark.hadoop.fs.s3a.committer.staging.tmp.path /tmp/staging
|
||||
spark.hadoop.fs.s3a.committer.staging.unique-filenames true
|
||||
spark.hadoop.fs.s3a.committer.threads 2048 # number of threads writing to MinIO
|
||||
spark.hadoop.fs.s3a.committer.threads 2048 # number of threads writing to Silo
|
||||
spark.hadoop.fs.s3a.connection.establish.timeout 5000
|
||||
spark.hadoop.fs.s3a.connection.maximum 8192 # maximum number of concurrent conns
|
||||
spark.hadoop.fs.s3a.connection.ssl.enabled false
|
||||
@@ -154,11 +154,11 @@ Navigate to **Services** -> **Hive** -> **CONFIGS**-> **ADVANCED** as shown belo
|
||||
|
||||
|
||||
|
||||
Navigate to “**Custom hive-site**” to configure MinIO parameters for `_s3a_` connector
|
||||
Navigate to “**Custom hive-site**” to configure Silo parameters for `_s3a_` connector
|
||||
|
||||
|
||||
|
||||
Add the following optimal entries for `hive-site.xml` to configure Hive with **MinIO**.
|
||||
Add the following optimal entries for `hive-site.xml` to configure Hive with **Silo**.
|
||||
|
||||
```
|
||||
hive.blobstore.use.blobstore.as.scratchdir=true
|
||||
@@ -188,7 +188,7 @@ Test the Spark installation by running the following compute intensive example,
|
||||
Follow these steps to run the Spark Pi example:
|
||||
|
||||
- Login as user **‘spark’**.
|
||||
- When the job runs, the library can now use **MinIO** during intermediate processing.
|
||||
- When the job runs, the library can now use **Silo** during intermediate processing.
|
||||
- Navigate to a node with the Spark client and access the spark2-client directory:
|
||||
|
||||
```
|
||||
@@ -224,7 +224,7 @@ WordCount is a simple program that counts how often a word occurs in a text file
|
||||
The following example submits WordCount code to the Scala shell. Select an input file for the Spark WordCount example. We can use any text file as input.
|
||||
|
||||
- Login as user **‘spark’**.
|
||||
- When the job runs, the library can now use **MinIO** during intermediate processing.
|
||||
- When the job runs, the library can now use **Silo** during intermediate processing.
|
||||
- Navigate to a node with Spark client and access the spark2-client directory:
|
||||
|
||||
```
|
||||
@@ -290,7 +290,7 @@ scala> counts.count()
|
||||
364
|
||||
```
|
||||
|
||||
To view the output from MinIO exit the Scala shell. View WordCount job status:
|
||||
To view the output from Silo exit the Scala shell. View WordCount job status:
|
||||
|
||||
```
|
||||
hadoop fs -ls s3a://testbucket/wordcount
|
||||
|
||||
Reference in New Issue
Block a user