This is the multi-page printable view of this section. .

Return to the regular view of this page.

MinIO

The open source object storage service

1: Usage
2: Configure
3: Parameter
4: Administration
5: Playbook
6: Monitor
7: FAQ

Min.IO: S3-Compatible Open-Source Multi-Cloud Object Storage designed to be scalable, secure, and handy. It has native multi-node multi-driver HA support and can store documents, pictures, videos, and backups. It’s an optional module in Pigsty.

You can use MinIO as an optional PostgreSQL backup storage repo, in addition to the default local posix FS repo. If the MinIO repo is used, the MINIO module should be installed before any PGSQL modules. MinIO requires a trusted CA to work, so you have to install it after NODE.

Configuration

    Configure minio module, and use multiple minio nodes.

Parameters

    Customize minio components with 15 parameters

Administration

    Create, remove, expand, shrink, upgrade minio cluster

Playbooks

    Ansible playbooks that can be used in this module

Monitoring

    Dashboards, metrics, record & alerting rules.

Usage

    How to use the mcli and configure backup repo

1 - Usage

Get started with MinIO and MCli, how to access the MinIO service?

After MinIO cluster is configured and deployed with the playbook, you can start using and accessing the MinIO cluster by following the instructions here.

Deploy Cluster

It is straightforward to deploy a single-node MinIO instance with Pigsty.

minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

Define it in the config inventory, then run the playbook:

./minio.yml -l minio

The install.yml playbook will automatically create the MinIO cluster defined in the inventory, so you don’t need to run the minio.yml playbook manually, if you choose the default one-pass installation.

If you plan to deploy a production-grade large-scale multi-node MinIO cluster, we strongly recommend you to read the Pigsty MinIO configuration document and the MinIO document before proceeding.

Access Cluster

You have to access MinIO via HTTPS, so make sure the default minio service domain (sss.pigsty) point to the right place:

You can add static resolution records in node_etc_hosts or manually modify the /etc/hosts file
You can add a record on the internal DNS server if you are using DNS service
You can add a record in dns_records if you are using the DNSMASQ on infra nodes

It is recommended to use the first method: static DNS resolution records to avoid MinIO’s additional dependency on DNS in production environments.

You have to point the MinIO service domain to the IP address and service port of the MinIO server node, or the IP address and service port of the load balancer. Pigsty will use the default domain name sss.pigsty and default port 9000.

For example, if you are using haproxy to expose MinIO service like this, the port may be 9002.

Adding Alias

To access the MinIO server cluster using the mcli client, you need to configure the server alias first:

mcli alias ls  # list minio alias (the default is sss)
mcli alias set sss https://sss.pigsty:9000 minioadmin minioadmin              # root user
mcli alias set sss https://sss.pigsty:9002 minioadmin minioadmin              # root user, on load balancer port 9002

mcli alias set pgbackrest https://sss.pigsty:9000 pgbackrest S3User.Backup    # use another user

There’s a pre-configured MinIO alias named sss on the admin user of the admin node, you can use it directly.

For the full functionality of the MinIO client tool mcli, please refer to the documentation: MinIO Client.

Manage User

You can manage biz users in MinIO using mcli, for example, you can create the two default biz users using the command line:

mcli admin user list sss     # list all users
set +o history               # hide shell history
mcli admin user add sss dba S3User.DBA
mcli admin user add sss pgbackrest S3User.Backup
set -o history

Manage Bucket

You can manage bucket with mcli:

mcli ls sss/                         # list all bucket on 'sss'
mcli mb --ignore-existing sss/hello  # create a bucket named 'hello'
mcli rb --force sss/hello            # delete the 'hello' bucket

Mange Object

You can perform object CRUD with cli, for example:

mcli cp /www/pigsty/* sss/infra/     # upload local repo content to infra bucket
mcli cp sss/infra/plugins.tgz /tmp/  # download file to local from minio
mcli ls sss/infra                    # list all files in the infra bucket
mcli rm sss/infra/plugins.tgz        # delete file in infra bucket
mcli cat sss/infra/repo_complete     # output the content of

Check the Tutorial: Object Management for detail

Use rclone

Pigsty repo has rclone available, a convenient cloud object storage client that you can use to access MinIO services.

yum install rclone; # el compatible
dnf install rclone; # debian/ubuntu

mkdir -p ~/.config/rclone/;
tee ~/.config/rclone/rclone.conf > /dev/null <<EOF
[sss]
type = s3
access_key_id = minioadmin
secret_access_key = minioadmin
endpoint = sss.pigsty:9000
EOF

rclone ls sss:/

Backup Repo

The MinIO is used as a backup repository for pgBackRest by default in Pigsty. When you modify the pgbackrest_method to minio, the PGSQL module will automatically switch the backup repository to MinIO.

pgbackrest_method: local          # pgbackrest repo method: local,minio,[user-defined...]
pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix fs
    path: /pg/backup              # local backup directory, `/pg/backup` by default
    retention_full_type: count    # retention full backups by count
    retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # minio is s3-compatible, so s3 is used
    s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
    s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
    s3_bucket: pgsql              # minio bucket name, `pgsql` by default
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    s3_uri_style: path            # use path style uri for minio rather than host style
    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    storage_port: 9000            # minio port, 9000 by default
    storage_ca_file: /pg/cert/ca.crt  # minio ca file path, `/pg/cert/ca.crt` by default
    bundle: y                     # bundle small files into a single file
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retention full backup by time on minio repo
    retention_full: 14            # keep full backup for last 14 days

Beware that if you are using MinIO through load balancer, you should use the corresponding domain name and port number here.

2 - Configure

Define SNSD, SNMD, MNMD MinIO Clusters

You have to define a MinIO cluster in the config inventory before deploying it.

There are 3 major deployment modes for MinIO clusters:

SNSD: Single-Node Single-Drive: you can use any dir as a disk driver in this mode, convenient for dev/test/demo.
SNMD: Single-Node Multi-Drive: a compromise mode, use multiple disks (>=2) on a single server, only when resources are extremely limited.
MNMD: Multi-Node Multi-Drive: standard production deployment with the best reliability, but requires multiple servers and real drivers.

We recommend using SNSD and MNMD for development and production deployment, respectively, and SNMD only when resources are extremely limited (only one server).

Besides, you can use multi-pool deployment to scale an existing MinIO cluster, or directly deploy multiple clusters.

When using a multi-node MinIO cluster, you can access the service from any node, so the best practice is to use a load balancer and HA access.

Core Param

There’s one and only one core param for MinIO deployment, which is MINIO_VOLUMES, which specifies the nodes, drivers, pools of a minio cluster

Pigsty will auto-generate MINIO_VOLUMES according to the config inventory for you, but you can always override it directly. If not explicitly specified, Pigsty will generate it according to the following rules:

SNSD: MINIO_VOLUMES points to any dir on local node, from minio_data
SNMD: MINIO_VOLUMES points to a series of real drivers on local node, from minio_data
MNMD: MINIO_VOLUMES points to multiple nodes & multiple drivers, according to minio_data and minio_node
Use minio_data to specify drivers on each node, such as /data{1...4}
Use minio_node to specify node name pattern, such as ${minio_cluster}-${minio_seq}.pigsty
Multi-Pool: MINIO_VOLUMES need to be explicitly specified

Single-Node Single-Drive

Tutorial: deploy-minio-single-node-single-drive

To define a singleton MinIO instance, it’s straightforward:

# 1 Node 1 Driver (DEFAULT)
minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

The only required params are minio_seq and minio_cluster, which generate a unique identity for each MinIO instance.

Single-Node Single-Driver mode is for dev purposes, so you can use a common dir as the data dir. The default data dir for SNSD minio is specified by minio_data, which is /data/minio by default. Beware that in multi-driver or multi-node mode, MinIO will refuse to start if using a common dir as the data dir rather than a mount point.

We strongly recommend using a static domain name record to access MinIO. For example, the default sss.pigsty if minio_domain can be added to all nodes through:

node_etc_hosts: ["10.10.10.10 sss.pigsty"] # domain name to access minio from all nodes (required)

Single-Node Multi-Drive

Reference: deploy-minio-single-node-multi-drive

To use multiple disks on a single node, you have to specify the minio_data in the format of {{ prefix }}{x...y}, which defines a series of disk mount points.

minio:
  hosts: { 10.10.10.10: { minio_seq: 1 } }
  vars:
    minio_cluster: minio         # minio cluster name, minio by default
    minio_data: '/data{1...4}'   # minio data dir(s), use {x...y} to specify multi drivers

Use real drivers and mountpoint

Beware that in multi-driver or multi-node mode, MinIO will refuse to start if using a common dir as the data dir rather than a mount point.

This example defines a single-node MinIO cluster with 4 drivers: /data1, /data2, /data3, /data4. You have to mount them properly before launching MinIO:

The vagrant MinIO sandbox has a pre-defined 4-node MinIO cluster with 4 drivers. You have to properly mount them before starting MinIO (be sure to format disks with xfs):

mkfs.xfs /dev/vdb; mkdir /data1; mount -t xfs /dev/sdb /data1;
mkfs.xfs /dev/vdc; mkdir /data2; mount -t xfs /dev/sdb /data2;
mkfs.xfs /dev/vdd; mkdir /data3; mount -t xfs /dev/sdb /data3;
mkfs.xfs /dev/vde; mkdir /data4; mount -t xfs /dev/sdb /data4;

Disk management is beyond this topic, just make sure your /etc/fstab is properly configured to auto-mount disks after reboot.

/dev/vdb /data1 xfs defaults,noatime,nodiratime 0 0
/dev/vdc /data2 xfs defaults,noatime,nodiratime 0 0
/dev/vdd /data3 xfs defaults,noatime,nodiratime 0 0
/dev/vde /data4 xfs defaults,noatime,nodiratime 0 0

SNMD mode can utilize multiple disks on a single server to provide higher performance and capacity, and tolerate partial disk failures.

But it can do nothing with node failure, and you can’t add new nodes at runtime, so we don’t recommend using SNMD mode in production unless you have a special reason.

Multi-Node Multi-Drive

Reference: deploy-minio-multi-node-multi-drive

The extra minio_node param will be used for a multi-node deployment in addition to the minio_data

For example, this configuration defines a 4-node MinIO cluster with 4 drivers per node:

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }  # nodename: minio-1.pigsty
    10.10.10.11: { minio_seq: 2 }  # nodename: minio-2.pigsty
    10.10.10.12: { minio_seq: 3 }  # nodename: minio-3.pigsty
    10.10.10.13: { minio_seq: 4 }  # nodename: minio-4.pigsty
  vars:
    minio_cluster: minio
    minio_data: '/data{1...4}'                         # 4-disk per node
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio name pattern

The minio_node param specifies the MinIO node name pattern, which is ${minio_cluster}-${minio_seq}.pigsty by default. The server name is very important for MinIO to identify and access other nodes in the cluster. It will be populated with minio_cluster and minio_seq, and write to /etc/hosts of all minio cluster members.

In this case, the MINIO_VOLUMES will be set to https://minio-{1...4}.pigsty/data{1...4} to identify the 16 disks on 4 nodes.

Multi-Pool

MinIO’s architecture allows for cluster expansion by adding new storage pools. In Pigsty, you can achieve this by explicitly specifying the minio_volumes param to specify nodes/disks for each pool.

For example, suppose you have already created a MinIO cluster as defined in the Multi-Node Multi-Disk example, and now you want to add a new storage pool consisting of four nodes.

You can specify minio_volumes here to allocate nodes for each pool to scale out the cluster.

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }
    10.10.10.11: { minio_seq: 2 }
    10.10.10.12: { minio_seq: 3 }
    10.10.10.13: { minio_seq: 4 }

    10.10.10.14: { minio_seq: 5 }
    10.10.10.15: { minio_seq: 6 }
    10.10.10.16: { minio_seq: 7 }
    10.10.10.17: { minio_seq: 8 }
  vars:
    minio_cluster: minio
    minio_data: "/data{1...4}"
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio 节点名称规则
    minio_volumes: 'https://minio-{1...4}.pigsty:9000/data{1...4} https://minio-{5...8}.pigsty:9000/data{1...4}'

Here, the two space-separated parameters represent two storage pools, each with four nodes and four disks per node.

For more information on storage pools, please refer to Management Plan: MinIO Cluster Expansion.

Multiple Clusters

You can deploy new MinIO nodes as a completely new MinIO cluster by defining a new group with a different cluster name.

The following configuration declares two independent MinIO clusters:

minio1:
  hosts:
    10.10.10.10: { minio_seq: 1 }
    10.10.10.11: { minio_seq: 2 }
    10.10.10.12: { minio_seq: 3 }
    10.10.10.13: { minio_seq: 4 }
  vars:
    minio_cluster: minio2
    minio_data: "/data{1...4}"

minio2:
  hosts:
    10.10.10.14: { minio_seq: 5 }
    10.10.10.15: { minio_seq: 6 }
    10.10.10.16: { minio_seq: 7 }
    10.10.10.17: { minio_seq: 8 }
  vars:
    minio_cluster: minio2
    minio_data: "/data{1...4}"
    minio_alias: sss2
    minio_domain: sss2.pigsty
    minio_endpoint: sss2.pigsty:9000

Please note that by default, Pigsty allows only one MinIO cluster per deployment. If you need to deploy multiple MinIO clusters, some parameters with default values need to be explicitly set and cannot be omitted to avoid naming conflicts, as shown above.

Expose Service

MinIO will serve on port 9000 by default. If a multi-node MinIO cluster is deployed, you can access its service via any node. It would be better to expose MinIO service via a load balancer, such as the default haproxy on NODE, or use the L2 vip.

To expose MinIO service with haproxy, you have to define an extra service with haproxy_services:

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
    10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
    10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
  vars:
    minio_cluster: minio
    node_cluster: minio
    minio_data: '/data{1...2}'         # use two disk per node
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio node name pattern
    haproxy_services:                  # EXPOSING MINIO SERVICE WITH HAPROXY
      - name: minio                    # [REQUIRED] service name, unique
        port: 9002                     # [REQUIRED] service port, unique
        options:                       # [OPTIONAL] minio health check
          - option httpchk
          - option http-keep-alive
          - http-check send meth OPTIONS uri /docs/minio/health/live
          - http-check expect status 200
        servers:
          - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

MinIO uses port 9000 by default. A multi-node MinIO cluster can be accessed by connecting to any one of its nodes.

Service access falls under the scope of the NODE module, and we’ll provide only a basic introduction here.

High-availability access to a multi-node MinIO cluster can be achieved using an L2 VIP or HAProxy. For example, you can use Keepalived to bind an L2 VIP to the MinIO cluster, or use the haproxy component provided by the NODE module to expose MinIO services through a load balancer.

# minio cluster with 4 nodes and 4 drivers per node
minio:
  hosts:
    10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
    10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
    10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
    10.10.10.13: { minio_seq: 4 , nodename: minio-4 }
  vars:
    minio_cluster: minio
    minio_data: '/data{1...4}'
    minio_buckets: [ { name: pgsql }, { name: infra }, { name: redis } ]
    minio_users:
      - { access_key: dba , secret_key: S3User.DBA, policy: consoleAdmin }
      - { access_key: pgbackrest , secret_key: S3User.SomeNewPassWord , policy: readwrite }

    # bind a node l2 vip (10.10.10.9) to minio cluster (optional)
    node_cluster: minio
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.9
    vip_interface: eth1

    # expose minio service with haproxy on all nodes
    haproxy_services:
      - name: minio                    # [REQUIRED] service name, unique
        port: 9002                     # [REQUIRED] service port, unique
        balance: leastconn             # [OPTIONAL] load balancer algorithm
        options:                       # [OPTIONAL] minio health check
          - option httpchk
          - option http-keep-alive
          - http-check send meth OPTIONS uri /docs/minio/health/live
          - http-check expect status 200
        servers:
          - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

In the configuration above, HAProxy is enabled on all nodes of the MinIO cluster, exposing MinIO services on port 9002, and a Layer 2 VIP is bound to the cluster. When in use, users should point the sss.pigsty domain name to the VIP address 10.10.10.9 and access MinIO services using port 9002. This ensures high availability, as the VIP will automatically switch to another node if any node fails.

In this scenario, you may also need to globally modify the destination of domain name resolution and adjust the minio_endpoint parameter to change the endpoint address corresponding to the MinIO alias on the management node:

minio_endpoint: https://sss.pigsty:9002   # Override the default https://sss.pigsty:9000
node_etc_hosts: ["10.10.10.9 sss.pigsty"] # Other nodes will use the sss.pigsty domain

Dedicate Proxies

Pigsty allows using dedicate load balancer cluster instead of the node cluster itself to run VIP & HAProxy.

For example, the prod template uses this way.

proxy:
  hosts:
    10.10.10.18 : { nodename: proxy1 ,node_cluster: proxy ,vip_interface: eth1 ,vip_role: master }
    10.10.10.19 : { nodename: proxy2 ,node_cluster: proxy ,vip_interface: eth1 ,vip_role: backup }
  vars:
    vip_enabled: true
    vip_address: 10.10.10.20
    vip_vrid: 20

    haproxy_services:      # expose minio service : sss.pigsty:9000
      - name: minio        # [REQUIRED] service name, unique
        port: 9000         # [REQUIRED] service port, unique
        balance: leastconn # Use leastconn algorithm and minio health check
        options: [ "option httpchk", "option http-keep-alive", "http-check send meth OPTIONS uri /docs/minio/health/live", "http-check expect status 200" ]
        servers:           # reload service with ./node.yml -t haproxy_config,haproxy_reload
          - { name: minio-1 ,ip: 10.10.10.21 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.22 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.23 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-4 ,ip: 10.10.10.24 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-5 ,ip: 10.10.10.25 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

In this case, you need to manually configure the DNS resolution to point sss.pigsty to the VIP address of dedicated proxies cluster

minio_endpoint: https://sss.pigsty:9002    # overwrite the defaults: https://sss.pigsty:9000
node_etc_hosts: ["10.10.10.20 sss.pigsty"] # domain name to access minio from all nodes (required)

Access Service

To use the exposed service, you have to update/append the MinIO credential in the pgbackrest_repo section:

# This is the newly added HA MinIO Repo definition, USE THIS INSTEAD!
minio_ha:
  type: s3
  s3_endpoint: minio-1.pigsty   # s3_endpoint could be any load balancer: 10.10.10.1{0,1,2}, or domain names point to any of the 3 nodes
  s3_region: us-east-1          # you could use external domain name: `sss.pigsty` ,which resolve to any members (`minio_domain`)
  s3_bucket: pgsql              # instance & nodename can be used : minio-1.pigsty minio-1.pigsty minio-1.pigsty minio-1 minio-2 minio-3
  s3_key: pgbackrest            # Betters using a new password for MinIO pgbackrest user
  s3_key_secret: S3User.SomeNewPassWord
  s3_uri_style: path
  path: /pgbackrest
  storage_port: 9002            # Use the load balancer port 9002 instead of default 9000 (direct access)
  storage_ca_file: /etc/pki/ca.crt
  bundle: y
  cipher_type: aes-256-cbc      # Better using a new cipher password for your production environment
  cipher_pass: pgBackRest.With.Some.Extra.PassWord.And.Salt.${pg_cluster}
  retention_full_type: time
  retention_full: 14

Expose Console

MinIO has a built-in console that can be accessed via HTTPS @ minio_admin_port. If you want to expose the MinIO console to the outside world, you can add MinIO to infra_portal.

# ./infra.yml -t nginx
infra_portal:
  home         : { domain: h.pigsty }
  grafana      : { domain: g.pigsty ,endpoint: "${admin_ip}:3000" , websocket: true }
  prometheus   : { domain: p.pigsty ,endpoint: "${admin_ip}:9058" }
  alertmanager : { domain: a.pigsty ,endpoint: "${admin_ip}:9059" }
  blackbox     : { endpoint: "${admin_ip}:9115" }
  loki         : { endpoint: "${admin_ip}:3100" }

  # MinIO console require HTTPS / Websocket to work
  minio        : { domain: m.pigsty     ,endpoint: "10.10.10.10:9001" ,scheme: https ,websocket: true }
  minio10      : { domain: m10.pigsty   ,endpoint: "10.10.10.10:9001" ,scheme: https ,websocket: true }
  minio11      : { domain: m11.pigsty   ,endpoint: "10.10.10.11:9001" ,scheme: https ,websocket: true }
  minio12      : { domain: m12.pigsty   ,endpoint: "10.10.10.12:9001" ,scheme: https ,websocket: true }
  minio13      : { domain: m13.pigsty   ,endpoint: "10.10.10.13:9001" ,scheme: https ,websocket: true }

Beware that MinIO console should be accessed via HTTPS, please DO NOT expose MinIO console without encryption in production.

Which means you usually need to add m.pigsty resolution to your DNS server, or /etc/hosts on your local host, to access the MinIO console.

Meanwhile, if you are using Pigsty’s self-signed CA rather than a regular public CA, you usually need to manually trust the CA or certificate to skip the “insecure” warning in the browser.

3 - Parameter

Customize MinIO parameters

MinIO is a S3 compatible object storage service. Which is used as an optional central backup storage repo for PostgreSQL.

You may also use it for other purposes, such as storing large files, documents, pictures & videos.

Parameters

There are 21 documented settings: 16 declared MINIO parameters, 2 derived values that may be overridden, and 3 MINIO_REMOVE flags.

Parameter	Type	Level	Comment
`minio_seq`	int	I	minio instance identifier, REQUIRED
`minio_cluster`	string	C	minio cluster name, minio by default
`minio_user`	username	C	minio os user, `minio` by default
`minio_https`	bool	G	use https for minio, true by default
`minio_node`	string	C	minio node name pattern
`minio_data`	path	C	minio data dir(s), use `{x...y}` to specify multi drivers
`minio_volumes`	string	C	minio core parameter, specify nodes and disks, auto-gen by default
`minio_domain`	string	G	minio external domain name, `sss.pigsty` by default
`minio_port`	port	C	minio service port, 9000 by default
`minio_admin_port`	port	C	minio console port, 9001 by default
`minio_access_key`	username	C	root access key, `minioadmin` by default
`minio_secret_key`	password	C	root secret key, `minioadmin` by default
`minio_extra_vars`	string	C	extra environment variables for minio server
`minio_provision`	bool	G/C	run minio provisioning tasks?
`minio_alias`	string	G	alias name for local minio deployment
`minio_endpoint`	string	C	corresponding host:port for above minio alias
`minio_buckets`	bucket[]	C	list of minio bucket to be created
`minio_users`	user[]	C	list of minio user to be created

MINIO_REMOVE Parameters (for minio_remove role)

Parameter	Type	Level	Comment
`minio_safeguard`	bool	G/C/A	prevent accidental removal? (default: false)
`minio_rm_data`	bool	G/C/A	remove minio data during removal? (default: true)
`minio_rm_pkg`	bool	G/C/A	uninstall minio packagkes during removal? (default: false)

The minio_volumes and minio_endpoint are auto-generated parameters, but you can explicitly override these two parameters.

Defaults

MINIO: 18 settings (including 2 derived values), defined in roles/minio/defaults/main.yml

#-----------------------------------------------------------------
# MINIO
#-----------------------------------------------------------------
#minio_seq: 1                     # minio instance identifier, REQUIRED
minio_cluster: minio              # minio cluster identifier, REQUIRED
minio_user: minio                 # minio os user, `minio` by default
minio_https: true                 # use https for minio, true by default
minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio node name pattern
minio_data: '/data/minio'         # minio data dir(s), use {x...y} to specify multi drivers
#minio_volumes:                   # minio data volumes, override defaults if specified
minio_domain: sss.pigsty          # minio external domain name, `sss.pigsty` by default
minio_port: 9000                  # minio service port, 9000 by default
minio_admin_port: 9001            # minio console port, 9001 by default
minio_access_key: minioadmin      # root access key, `minioadmin` by default
minio_secret_key: minioadmin      # root secret key, `minioadmin` by default
minio_extra_vars: ''              # extra environment variables
minio_provision: true             # run minio provisioning tasks?
minio_alias: sss                  # alias name for local minio deployment
#minio_endpoint: https://sss.pigsty:9000 # if not specified, overwritten by defaults
minio_buckets:                    # list of minio bucket to be created
  - { name: pgsql }
  - { name: meta ,versioning: true }
  - { name: data }
minio_users:                      # list of minio user to be created
  - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
  - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
  - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

MINIO_REMOVE: 3 Parameters, defined in roles/minio_remove/defaults/main.yml:

#-----------------------------------------------------------------
# MINIO_REMOVE
#-----------------------------------------------------------------
minio_safeguard: false            # prevent accidental removal?
minio_rm_data: true               # remove minio data during removal?
minio_rm_pkg: false               # uninstall minio packages during removal?

`minio_seq`

minio instance identifier, REQUIRED identity parameters. no default value, you have to assign it manually

`minio_cluster`

minio cluster name, minio by default. This is useful when deploying multiple MinIO clusters

`minio_user`

minio os user name, minio by default

`minio_https`

Use HTTPS or HTTP for MinIO service, true by default, means using HTTPS.

Beware that pgbackrest requires MinIO HTTPS to work properly, but if you don’t use minio for that, and don’t want to use HTTPS for MinIO, you can set this to false.

`minio_node`

minio node name pattern, this is used for multi-node deployment

default values: ${minio_cluster}-${minio_seq}.pigsty

`minio_data`

minio data dir(s)

default values: /data/minio, which is a common dir for single-node deployment.

For a multi-drive deployment, you can use {x...y} notion to specify multi drivers.

`minio_volumes`

The only core parameter of MinIO, if not specified, it will be auto-generated by the following rule:

minio_volumes: "{% if minio_cluster_size|int > 1 %}https://{{ minio_node|replace('${minio_cluster}', minio_cluster)|replace('${minio_seq}',minio_seq_range) }}:{{ minio_port|default(9000) }}{% endif %}{{ minio_data }}"

In case of SNSD or SNMD deployment, minio_volumes directly uses the value of minio_data
In case of MNMD deployment, minio_volumes uses the values of minio_node, minio_port, minio_data to generate this param:
In case of multiple storage pool, you have to override minio_volumes to specify multiple node pools explicitly.

It user’s responsibility to make sure the parameters used in minio_volumes are consistent with minio_node, minio_port, minio_data.

`minio_domain`

minio service domain name, sss.pigsty by default.

The client can access minio S3 service via this domain name. This name will be registered to local DNSMASQ and included in SSL certs.

`minio_port`

minio service port, 9000 by default

`minio_admin_port`

minio console port, 9001 by default

`minio_access_key`

root access key, minioadmin by default

`minio_secret_key`

root secret key, minioadmin by default

default values: minioadmin

Change PASSWORDS!

It’s very important to change this password in your deployment!

`minio_extra_vars`

extra environment variables for minio server. Check Minio Server for the complete list.

default value is empty string, you can use multiline string to passing multiple environment variables.

`minio_alias`

MinIO alias name for the local MinIO cluster

default values: sss, which will be written to infra nodes’ / admin users’ client alias profile.

`minio_endpoint`

The corresponding host:port for the above MinIO alias. This parameter is not defined by default.

If not defined, it will be overwritten by the following default value:

mcli alias set {{ minio_alias }} {% if minio_endpoint is defined and minio_endpoint != '' %}{{ minio_endpoint }}{% else %}https://{{ minio_domain }}:{{ minio_port }}{% endif %} {{ minio_access_key }} {{ minio_secret_key }}

This alias & endpoint will be added to the admin user on the admin node.

`minio_buckets`

list of minio bucket to be created by default:

minio_buckets:                    # list of minio bucket to be created
  - { name: pgsql }
  - { name: meta ,versioning: true }
  - { name: data }

Three default buckets are created by default, with different policies.

The pgsql bucket is used for PostgreSQL backups by default. while meta and data are open buckets for other purposes. For example, the supabase template may use the data bucket to store business data.

And if you have important metadata that requires versioning, you can use the meta bucket out-of-the-box.

Every bucket will have a corresponding policy, with names same as the bucket name. e.g., the pgsql policy have all the privileges on the pgsql bucket, and so on.

You can also add the lock flag to bucket definition, which will enable the object locking feature to prevent accidental deletion of objects in the bucket.

`minio_users`

list of minio users to be created, default value:

minio_users:                      # list of minio user to be created
  - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
  - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
  - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

Two default users are created for PostgreSQL DBA and pgBackREST.

Change PASSWORDS!

Please change these passwords in serious production deployments.

`minio_safeguard`

prevent accidental removal? default value is false

If enabled, the minio-rm.yml playbook will abort and refuse to remove the MinIO cluster, providing protection against accidental deletions.

`minio_rm_data`

remove minio data during removal? default value is true

When enabled, the minio-rm.yml playbook will remove MinIO data directories and configuration files during cluster removal.

`minio_rm_pkg`

uninstall minio packages during removal? default value is false

When enabled, the minio-rm.yml playbook will uninstall MinIO packages during cluster removal. This is disabled by default to preserve the MinIO installation for potential future use.

4 - Administration

Create, remove, expand, shrink, upgrade minio cluster

Here are some administration SOP for MinIO:

Create Cluster
Remove Cluster
Expand Cluster
Shrink Cluster
Upgrade Cluster
Node Failure Recovery
Disk failure Recovery

Check ETCD: FAQ for more questions.

Create Cluster

To create a MinIO cluster, define the minio cluster in inventory first:

minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

The minio_cluster param mark this cluster as a MinIO cluster, and the minio_seq is the sequence number of the MinIO node, which is used to generate MinIO node name like minio-1, minio-2, etc.

This snippet defines a single-node MinIO cluster, using the following command to create the MinIO cluster:

./minio.yml -l minio  # init MinIO module on the minio group

Remove Cluster

To destroy an existing MinIO cluster, use the dedicated minio-rm.yml playbook:

./minio-rm.yml -l minio                                  # Remove MinIO cluster

You can also customize the removal process with parameters:

./minio-rm.yml -l minio -e minio_rm_pkg=true            # Also remove packages
./minio-rm.yml -l minio -e minio_rm_data=false          # Keep data directories
./minio-rm.yml -l minio -e minio_safeguard=true         # Enable safeguard (will abort)

Legacy approach (deprecated):

Architecture Change: Pigsty v3.6+

Since Pigsty v3.6+, MinIO cluster removal has been moved to the dedicated minio-rm.yml playbook using the minio_remove role. The prometheus monitoring targets are automatically cleaned up during the removal process.

Expand Cluster

Tutorial: Expand MinIO Deployment

You cannot scale MinIO at node/disk level, but you can scale at storage pool (multiple nodes) level.

Assume you have a 4-node MinIO cluster and want to double the capacity by adding another four-node storage pool.

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
    10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
    10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
    10.10.10.13: { minio_seq: 4 , nodename: minio-4 }
  vars:
    minio_cluster: minio
    minio_data: '/data{1...4}'
    minio_buckets: [ { name: pgsql }, { name: infra }, { name: redis } ]
    minio_users:
      - { access_key: dba , secret_key: S3User.DBA, policy: consoleAdmin }
      - { access_key: pgbackrest , secret_key: S3User.SomeNewPassWord , policy: readwrite }

    # bind a node l2 vip (10.10.10.9) to minio cluster (optional)
    node_cluster: minio
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.9
    vip_interface: eth1

    # expose minio service with haproxy on all nodes
    haproxy_services:
      - name: minio                    # [REQUIRED] service name, unique
        port: 9002                     # [REQUIRED] service port, unique
        balance: leastconn             # [OPTIONAL] load balancer algorithm
        options:                       # [OPTIONAL] minio health check
          - option httpchk
          - option http-keep-alive
          - http-check send meth OPTIONS uri /docs/minio/health/live
          - http-check expect status 200
        servers:
          - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

Step 1, add 4 node definitions in the group, allocate sequence number 5 to 8. The key step is to modify the minio_volumes param, assign the new 4 nodes to a new storage pool.

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
    10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
    10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
    10.10.10.13: { minio_seq: 4 , nodename: minio-4 }
    # new nodes
    10.10.10.14: { minio_seq: 5 , nodename: minio-5 }
    10.10.10.15: { minio_seq: 6 , nodename: minio-6 }
    10.10.10.16: { minio_seq: 7 , nodename: minio-7 }
    10.10.10.17: { minio_seq: 8 , nodename: minio-8 }

  vars:
    minio_cluster: minio
    minio_data: '/data{1...4}'
    minio_volumes: 'https://minio-{1...4}.pigsty:9000/data{1...4} https://minio-{5...8}.pigsty:9000/data{1...4}'  # 新增的集群配置
    # misc params

Step 2, adding these nodes to Pigsty:

./node.yml -l 10.10.10.14,10.10.10.15,10.10.10.16,10.10.10.17

Step 3, Provisioning MinIO on new nodes with minio_install subtask (user, dir, pkg, …):

./minio.yml -l 10.10.10.14,10.10.10.15,10.10.10.16,10.10.10.17 -t minio_install

Step 4: Reconfigure the entire MinIO cluster on the whole cluster with minio_config subtask

./minio.yml -l minio -t minio_config

That is to say, the existing 4-nodes’ MINIO_VOLUMES configuration will be updated, too

Step 5: Restart the entire MinIO cluster simultaneously (be careful, do not rolling restart!):

./minio.yml -l minio -t minio_launch -f 10   # with 10 parallel

Step 6: This is optional, if you are using a load balancer, make sure the load balancer configuration is updated.

For example, add the new four nodes to the load balancer configuration:

# expose minio service with haproxy on all nodes
haproxy_services:
  - name: minio                    # [REQUIRED] service name, unique
    port: 9002                     # [REQUIRED] service port, unique
    balance: leastconn             # [OPTIONAL] load balancer algorithm
    options:                       # [OPTIONAL] minio health check
      - option httpchk
      - option http-keep-alive
      - http-check send meth OPTIONS uri /docs/minio/health/live
      - http-check expect status 200
    servers:
      - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

      - { name: minio-5 ,ip: 10.10.10.14 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-6 ,ip: 10.10.10.15 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-7 ,ip: 10.10.10.16 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-8 ,ip: 10.10.10.17 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

Then run the haproxy subtask of the node.yml playbook to update the load balancer configuration:

./node.yml -l minio -t haproxy_config,haproxy_reload   # re-configure and reload haproxy service definition

If node L2 VIP is also used to ensure reliable load balancer access, you also need to add new nodes (if any) to the existing NODE VIP group:

./node.yml -l minio -t node_vip  # reload node l2 vip configuration

Shrink Cluster

MinIO cannot scale down at the node/disk level, but you can retire at the storage pool (multiple nodes) level —— Add a new storage pool, drain the old storage pool, migrate to the new storage pool, and then retire the old storage pool.

Tutorial: Decommission Server Pool

Upgrade Cluster

Tutorial: Upgrade MinIO Deployment

First, download the new version of the MinIO software package to the local software repository of the INFRA node:

minio:
- amd64: https://dl.min.io/server/minio/release/linux-amd64/
- arm64: https://dl.min.io/server/minio/release/linux-arm64/
mcli:
- amd64: https://dl.min.io/client/mc/release/linux-amd64/
- arm64: https://dl.min.io/client/mc/release/linux-arm64/

and then rebuild the software repo with:

./infra.yml -t repo_create

You can upgrade all MinIO software packages with Ansible package module:

ansible minio -m package -b -a 'name=minio state=latest'  # upgrade MinIO server
ansible minio -m package -b -a 'name=mcli state=latest'   # upgrade mcli client

Finally, notify the MinIO cluster to restart with the mc command line tool:

mc admin service restart sss

Node Failure Recovery

Tutorial: Restore Hardware Failure Node

# 1. remove failure node
bin/node-rm <your_old_node_ip>

# 2. replace failure node with the same name (modify the inventory in case of IP change)
bin/node-add <your_new_node_ip>

# 3. provisioning MinIO on new node
./minio.yml -l <your_new_node_ip>

# 4. instruct MinIO to perform heal action
mc admin heal

Disk Failure Recovery

Tutorial: Restore Hardware Failure Disk

# 1. umount failure disk
umount /dev/<your_disk_device>

# 2. replace with new driver, format with xfs
mkfs.xfs /dev/sdb -L DRIVE1

# 3. don't forget to setup fstab for auto-mount
vi /etc/fstab
# LABEL=DRIVE1     /mnt/drive1    xfs     defaults,noatime  0       2

# 4. remount the new disk
mount -a

# 5. instruct MinIO to perform heal action
mc admin heal

5 - Playbook

control primitives

You have to configure minio cluster in the config inventory before running the playbook.

Playbook

There are two built-in playbooks for MinIO cluster management:

minio.yml for installing the MinIO cluster
minio-rm.yml for removing the MinIO cluster

`minio.yml`

minio.yml

minio-id : generate minio identity
minio_install : install minio/mcli
- minio_os_user : create os user minio
- minio_pkg : install minio/mcli package
- minio_dir : create minio directories
minio_config : generate minio config
- minio_conf : minio main config
- minio_cert : minio ssl cert
- minio_dns : write minio dns records
minio_launch : launch minio service
minio_register : register minio to prometheus
minio_provision : create minio aliases/buckets/users
- minio_alias : create minio client alias
- minio_bucket : create minio buckets
- minio_user : create minio biz users

Architecture Change: Pigsty v3.6+

Since Pigsty v3.6+, the minio.yml playbook and minio role are focused solely on cluster installation. All removal operations have been moved to the dedicated minio-rm.yml playbook using the minio_remove role.

Trusted ca file: /etc/pki/ca.crt should exist on all nodes already. which is generated in role: ca and loaded & trusted by default in role: node.

You should install MINIO module on Pigsty-managed nodes (i.e., Install NODE first)

`minio-rm.yml`

To remove the MinIO cluster, run the following playbook:

./minio-rm.yml    # remove MinIO cluster

Here are available sub tasks:

minio-id : generate minio identity for removal operations
minio_safeguard : safeguard against accidental removal (default: false)
prometheus : remove minio targets registration from prometheus
minio_svc : stop and disable minio service with systemd
minio_data : remove minio data (disable with minio_rm_data=false)
minio_pkg : uninstall minio packages (enable with minio_rm_pkg=true)

The removal playbook uses the new minio_remove role with configurable parameters:

minio_safeguard: Prevents accidental removal when set to true
minio_rm_data: Controls whether MinIO data is deleted (true by default)
minio_rm_pkg: Controls whether MinIO packages are uninstalled (false by default)

Commands

MINIO Playbook cheatsheet and common commands

./minio.yml -l <cls>                      # init MINIO module on group <cls>
./minio-rm.yml -l minio                   # remove MinIO cluster using dedicated removal playbook
./minio.yml -l minio -t minio_install     # install MinIO, setup dirs, without configure & launch
./minio.yml -l minio -t minio_config      # generate MinIO config & certs
./minio.yml -l minio -t minio_launch      # restart MinIO cluster

6 - Monitor

monitor MinIO clusters

Dashboard

There is one dashboard for MINIO module.

MinIO Overview: Overview of one single MinIO cluster

Alert Rules

There are 3 predefined alert rules for MinIO, defined in files/prometheus/rules/minio.yml

MinioServerDown
MinioNodeOffline
MinioDiskOffline

#==============================================================#
#                         Aliveness                            #
#==============================================================#
# MinIO server instance down
- alert: MinioServerDown
  expr: minio_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: minio }
  annotations:
    summary: "CRIT MinioServerDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      minio_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      http://g.pigsty/d/minio-overview

#==============================================================#
#                         Error                                #
#==============================================================#
# MinIO node offline triggers a p1 alert
- alert: MinioNodeOffline
  expr: avg_over_time(minio_cluster_nodes_offline_total{job="minio"}[5m]) > 0
  for: 3m
  labels: { level: 1, severity: WARN, category: minio }
  annotations:
    summary: "WARN MinioNodeOffline: {{ $labels.cls }} {{ $value }}"
    description: |
      minio_cluster_nodes_offline_total[cls={{ $labels.cls }}] = {{ $value }} > 0
      http://g.pigsty/d/minio-overview?from=now-5m&to=now&var-cls={{$labels.cls}}

# MinIO disk offline triggers a p1 alert
- alert: MinioDiskOffline
  expr: avg_over_time(minio_cluster_disk_offline_total{job="minio"}[5m]) > 0
  for: 3m
  labels: { level: 1, severity: WARN, category: minio }
  annotations:
    summary: "WARN MinioDiskOffline: {{ $labels.cls }} {{ $value }}"
    description: |
      minio_cluster_disk_offline_total[cls={{ $labels.cls }}] = {{ $value }} > 0
      http://g.pigsty/d/minio-overview?from=now-5m&to=now&var-cls={{$labels.cls}}

7 - FAQ

frequently asked questions

Fail to launch multi-node / multi-driver MinIO cluster.

In Multi-Driver or Multi-Node mode, MinIO will refuse to start if the data dir is not a valid mount point.

Use mounted disks for MinIO data dir rather than some regular directory. You can use the regular directory only in the single node, single drive mode.

How to deploy a multi-node multi-drive MinIO cluster?

Check Create Multi-Node Multi-Driver MinIO Cluster

How to add a member to the existing MinIO cluster?

You’d better plan the MinIO cluster before deployment… Since this requires a global restart

Check this: Expand MinIO Deployment

How to use a HA MinIO deployment for PGSQL?

Access the HA MinIO cluster with an optional load balancer and different ports.

Here is an example: Access MinIO Service