High-performance in-memory cache, optional data structure server
DOCKER
Container runtime, optional for running stateless app and tools
FERRET
MongoDB wire-protocol compatible on PostgreSQL, optional middleware
FAQ
What is Pigsty—and what isn't it?
Pigsty is a PostgreSQL distribution: an open-source, local-first RDS-like solution. It isn't a database (DBMS), but rather a management tool, solution, and set of best practices. Think of a database as a car, a DBA as the driver, RDS as a taxi service, and Pigsty as self-driving software.
What problem does Pigsty solve?
Database expertise is scarce and costly—you either hire expensive DBAs or pay a premium for cloud RDS. Now there's a new option: Pigsty (the "self-driving" equivalent). It lets you run databases without a dedicated DBA at under one-tenth the cost of typical RDS, while achieving greater performance and flexibility in a local cloud environment.
Why can Pigsty help you manage databases effectively?
Pigsty packages hard-earned best practices from large, complex PostgreSQL deployments into a reproducible solution. It tackles everything: scalable installs, high availability, connection pooling, monitoring, backup/recovery, parameter tuning, IaC-based cluster management, one-click deployment, and automated ops—letting you avoid common pitfalls and reinventing the wheel.
Why is Pigsty better than RDS?
Pigsty provides a richer feature set than most RDS offerings: 400+ extensions, support for 9+ Postgres kernels, a unique monitoring system, and proven architecture refined by top-tier enterprise use (Apple, Alibaba, Tantan). It's easy to use, deeply tested, and fueled by genuine passion—at a depth and maturity level typical RDS platforms can't match.
Why is Pigsty more cost-effective than RDS?
With Pigsty, you can run what's effectively a ¥400–¥1400/core/month RDS instance on bare-metal resources costing as little as ¥10/core/month—and skip paying a DBA's salary. At scale, total cost of ownership can drop by 90% or more. Pigsty slashes license, service, and personnel costs, so you can channel your budget where it really matters.
How does Pigsty help developers?
Pigsty integrates 400+ PG extensions for an all-in-Postgres approach—letting you replace Redis, Kafka, MySQL, Elasticsearch, vector databases, and even certain OLAP/big-data use cases with one stack. This boosts developer agility, cuts complexity, and enables self-serve DevOps without needing a dedicated DBA.
How does Pigsty help operations teams?
Pigsty's self-healing HA ensures you don't need to scramble when hardware fails—ops and DBAs can sleep easy. Advanced monitoring aids troubleshooting and performance tuning. IaC automates large-scale cluster management. Ops can double as DBAs, and DBAs skip heavy lifting to focus on high-value tasks—or just enjoy a tea break while learning PG.
Who created Pigsty?
Pigsty is primarily developed by @Vonng / Ruohang Feng, a PostgreSQL veteran with a decade of open-source contributions and production experience at Alibaba, Tantan, and Apple. He's the founder of a one-person company (YC China/MiraclePlus S22) offering professional postgres consultancy. He's also a top database/cloud KOL with over 100k followers.
What is Pigsty's ecosystem standing and impact?
Pigsty is one of the most active OSS projects in the PG ecosystem (OSSRank). It prevails in extension distribution and monitoring solutions, positioning itself as a challenger to cloud RDS. It's already in wide use across military, government, healthcare, internet, finance, and manufacturing sectors.
Which user scale is Pigsty suitable for?
Pigsty was built for massive PostgreSQL clusters, but it's also optimized for simplicity. Even solo developers or small businesses without DBA skills can get going easily. Largest known production scale: 25k vCPUs at 4.5 million QPS over six years. Smallest: a 1c1g VM demo or dev box.
Which capabilities does Pigsty offer?
Pigsty focuses on Postgres best practices and integrates well with complementary open-source tools like Etcd, Redis, MinIO, DuckDB, Prometheus, FerretDB, Babelfish, IvorySQL, PolarDB, OrioleDB, OpenHalo, Supabase, Greenplum, Dify, Odoo, and more.
Which scenarios does Pigsty fit?
Massive production postgres deployment, self-hosting RDS, object storage, caching, data warehousing, Supabase, etc. It can run enterprise apps like Odoo, Dify, wikis, GitLab, or serve as a monitoring hub for existing nodes and DBMS. It also excels at combining multiple PG extensions, powering interactive data apps, data visualization, and websites.
Is Pigsty open source and free?
Yes. Pigsty is 100% free software under the AGPLv3 license. You can use it freely for commercial purposes if you comply with that license. We value software freedom; for non-DBaaS/OEM use, we apply an Apache 2.0–like approach. See the license for details.
Does Pigsty offer commercial support?
Pigsty itself is free, but we provide commercial subscriptions for those who want assurance and expert support. Subscriptions extend coverage to more OS/PG/CPU architectures, plus consultancy. This delivers top-tier technical expertise and solutions, saving you time, reducing risk, and covering tough issues.
Can Pigsty be repackaged as multi-tenant DBaaS or rebranded?
Yes, under the AGPLv3 license—meaning any derivative work must be open-sourced. We reserve the right to pursue violations by public cloud/database vendors. If you prefer not to open-source derivatives, consider Pigsty Service Subscriptions, which provide explicit licensing and an AGPLv3 exemption for this scenario.
The configure script will generate the pigsty.yml config file inventory with good defaults according to your environment and input.
It’s OPTIONAL, you can edit the pigsty.yml directly as the tutorial shows.
There are many Config Templates for your reference, here are some quick examples:
./configure # use the default template, PG 18 with essential extensions./configure -v 17# default meta template with PG 17 instead of 18./configure -c rich # PG 18, local repo, download all extensions and install major ones./configure -c slim # minimal installation template, use with ./slim.yml playbook./configure -c app/supa # use the app/supa self-hosting supabase config template./configure -c ivory # use the ivorysql kernel instead of vanilla PG (pg18.0)./configure -i 10.11.12.13 # give primary IP address explicitly./configure -r china # use use china mirror instead of default repo./configure -c full -s # use the 4-node sandbox config template, without IP replace & probe
Let’s just do configure without any args, it may ask you for the primary IP if more than one is found.
[vagrant@node-2 pigsty]$ ./configure
configure pigsty v3.7.0 begin
[ OK ]region= default
[ OK ]kernel= Linux
[ OK ]machine= x86_64
[ OK ]package= rpm,dnf
[ OK ]vendor= rocky (Rocky Linux)[ OK ]version=9(9.6)[ OK ]sudo= vagrant ok
[ OK ]ssh=[email protected] ok
[WARN] Multiple IP address candidates found:
(1) 192.168.121.24 inet 192.168.121.24/24 brd 192.168.121.255 scope global dynamic noprefixroute eth0
(2) 10.10.10.12 inet 10.10.10.12/24 brd 10.10.10.255 scope global noprefixroute eth1
[ IN ] INPUT primary_ip address (of current meta node, e.g 10.10.10.10):
=> 10.10.10.12 # <------- INPUT YOUR PRIMARY IPV4 ADDRESS HERE![ OK ]primary_ip= 10.10.10.12 (from input)[ OK ]admin=[email protected] ok
[ OK ]mode= meta (el9)[ OK ]locale= C.UTF-8
[ OK ] configure pigsty doneproceed with ./install.yml
This script will replace the IP placeholder 10.10.10.10 to the primary IPv4 address of current node.
Beware of this when you are configure pigsty manually. Check the generated pigsty.yml to proceed.
Sometimes upstream repo (like linux / pgdg repo) may break, this do happen from time to time, and led to installation failure.
You can use pre-made offline packages to address this issue.
NEVER RUN THIS AGAIN ON EXISTING DEPLOYMENT!
Re-run this playbook entirely will nuke (wipe-out) the current deployment and create a new one!
If you have enough knowledge with ansible and know what you are doing, still do it with caution!
Once installed, you can explore the Interface and deploy More Nodes and more HA database clusters.
More
You can deploy & monitor More Clusters with pigsty: add definition to the Inventory and run:
You can access WebUI services from the following Ports directly (not recommended for prod).
Or use local / public Domain Names to access them via Nginx Portal.
SSL Certs can only be used with domains.
You can define business Database and User. These config templates has some pre-defined examples for your reference.
For example, the default meta config template has pre-defined a meta database with Pigsty CMDB schema (optional) and a vector extension.
It defines dbuser_meta as business admin user with DDL privilege, and a dbuser_view as read-only viewer user.
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }# <---- primary instance with read-write capabilityvars:pg_cluster:pg-meta # required identity parameter, usually same as group namepg_databases:# define business databases on this cluster, array of database definition- name:meta # REQUIRED, `name` is the only mandatory field of a database definitionbaseline:cmdb.sql # optional, database sql baseline path, (relative to files/)schemas:[pigsty ] # optional, additional schemas to be created, array of schema namesextensions:[vector ] # optional, additional extensions to be installed: array of `{name[,schema]}`comment:pigsty meta database # optional, comment string for this databasepg_users:# define business users/roles on this cluster, array of user definition- name:dbuser_meta # REQUIRED, `name` is the only mandatory field of a user definitionpassword:DBUser.Meta # optional, password can be a scram-sha-256 hash string or plain text- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly], comment:read-only viewer for meta database }pg_hba_rules:# example hba rules- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}
Which means you can also access the meta database with these two users:
In that case, your traffic is usually distributed by haproxy and pooled by pgbouncer before reaching the database.
Grafana
Grafana is the platform for monitoring and observability, it listens on port 3000 by default:
http://10.10.10.10:3000 (replace with your IP)
Access via domain name
Pigsty support static local domain (fake) for web components, you can access Grafana via http://g.pigsty through Nginx
It’s advised to use domain names, Since you can expose all services via Nginx with domain names, and use SSL certs for them.
Grafana Username and Password
Default credential: admin:pigsty. If you have changed the default credentials, use your own.
Just replace domain entries in infra_portal, and use make cert to apply for real certs for free
1.3 - Multi-Node
how to install pigsty on multiple nodes
There is a configuration Tutorial for expanding Pigsty from one node to multiple nodes.
While the easiest way is always pre-planing everything and provisioning them in one-pass.
1-node Setup
We already illustrated the 1-node installation in the Quick Start section, which may be the simplest setup.
It is not advised to put all the eggs in one basket, but even this one-node setup can be used for production,
as long as an external MinIO / S3 / NFS… is configured for PG as remote backup repo.
You can use the Vagrantsimu.rb to provision this environment.
You can run the entire simulation on a real server (72C / 256G) with libvirt as VM provider with vagrant.
2 infra nodes, monitoring each other
2 dedicated proxy nodes that run haproxy
5-node etcd cluster which tolerates 2 node failures, and 5-node redis sentinel cluster
redis standalone cluster redis-src and redis-dst, and native cluster redis-test
1.4 - Offline Install
install pigsty without Internet access?
Pigsty install from Internet upstream by default, while some environments are isolated from the Internet.
To address this issue, Pigsty supports offline installation from offline packages.
Step 1
Download the pigsty offline package, put it to `/tmp/pkg.tgz`
Step 2
Download the pigsty source package, extract it (assume `~/pigsty`)
Step 3
`cd ~/pigsty; ./bootstrap`, it will extract the pkg and use the local repo
Step 4
`vi ~/pigsty.yml`, overwrite [`node_repo_modules`](/docs/node/param#node_repo_modules) to `local` to use the local repo
Step 5
Run `./install.yml` as usual. it will install everything from the local repo.
What is offline package?
Offline package packs all the required RPM/DEB packages and their dependencies;
it is essentially a snapshot tarball of local APT / YUM repo, taken after a normal installation.
Offline Package is made for specific Linux OS Major.Minor Version
Using offline package on lower minor version may have a chance of success, but it is not guaranteed.
Using Offline Package
Place the offline package at /tmp/pkg.tgz, navigate to the ~/pigsty directory and execute ./bootstrap to extract and use the offline package.
Pigsty will extract it to /www/pigsty, configure the system repository list to enable this repository, and install ansible from it.
Since Pigsty v3.6, most configuration templates no longer build local software repositories by default, but install packages directly from Internet upstream.
A few configuration templates such as rich and full still retain the old behavior — building local repositories first before using them.
If you want to use the extracted and configured offline package in your own configuration, please modify the following settings:
repo_enabled: Set this to true to build local software repo (explicitly disabled in most templates)
node_repo_modules: Set this to local to install from local software repo for all nodes in the env
In most templates, this parameter is now explicitly configured as: node,infra,pgsql, which installs software packages directly from upstream repos.
Setting it to local will use the local repo alone, which is fastest and more reliable if applicable.
If you want to use both local and upstream repositories, you can set it to local,node,infra,pgsql
Pros and Cons
If you are using the exact same OS minor version above, using offline packages is advised.
Advantages of using offline package
Official offline packages are tested.
The easiest way to deliver on Internet isolated environment.
Speed up the installation process by pre-download all packages in one-pass.
The snapshot is ensured working without worrying about upstream dependency breakage.
Disadvantages of using offline package
If OS minor version not match, OS rpm/deb packages may break
It may not contain the latest updates and OS security patches.
Bootstrap
The bootstrap script will automatically detect /tmp/pkg.tgz and extract it to /www/pigsty by default.
It will also set up the os package manager repo file and install ansible and other utils.
Where are my repo files?
Bootstrap will wipe-out existing repo by default to ensure only the required repos are installed.
You can find them in /etc/yum.repos.d/backup (EL) or /etc/apt/backup (debian / ubuntu)
If you want to keep your existing repo
You can use the -k|--keep parameter to keep the existing repo files intact:
./bootstrap -k # or --keep
Make offline package
If your OS choice is not in the default offline package offering list,
you can make your own offline package with the built-in cache.yml playbook.
Step 1
Find a node running exact same OS version with the Internet access
Step 2
Run standard [online installation procedure](/docs/install) (use the `rich` template: `configure -c rich`)
Step 3
`cd ~/pigsty; ./cache.yml`: make and fetch the offline package to `dist/${version}/`
Step 4
Copy the offline package to the environment without Internet access (ftp, scp, usb,...)
Behavior Change in v3.6
Since Pigsty v3.6, most configuration templates install packages directly from Internet upstream, rather than downloading to the admin node first to build a local repository and then installing from it.
You can restore the previous default behavior by adjusting parameters, which is useful if you need to build your own offline packages:
repo_enabled: set to true to build local software repo
Some configuration templates, such as rich and full, still retain the old behavior — building local repositories first before using them, so no adjustment is needed.
We offer paid service to make offline packages for precise Linux major.minor versions. ($30)
Hybrid Approach
There’s a hybrid approach to using the offline package and do online install upon it.
This could be useful if your OS repo is a local Intranet repo.
Let’s say you are using RockyLinux 9.5, but the official offline package is made for RockyLinux 9.6.
You can use the el9 offline packages, and run the make repo-build before the ./install.yml,
to force Pigsty to perform a repo build task (infra.repo).
If you want to use the latest version or resolve any dependency conflict, you can remove these rpm/deb and run make repo-build,
pigsty will re-download them from the upstream repo.
1.5 - Minimal Install
install Postgres with minimal dependencies
If you just want HA PostgreSQL itself, without monitoring, infra, etc…, consider the slim installation.
ls /www/pigsty
cat ~/pigsty/pigsty.yml | grep node_repo_modules
sudo su - postgres
pg list
pb info
pg-backup incr
pb list
psql
SELECT * FROM pg_available_extensions;
# download this offline package and put it to /tmp/pkg.tgz, we just skip downloading here# curl https://github.com/pgsty/pigsty/releases/download/v3.6.0/pigsty-pkg-v3.6.0.el9.x86_64.tgz -o /tmp/pkg.tgzscp ~/pigsty/dist/v3.6.0/pigsty-v3.6.0.tgz meta:~/pigsty.tgz
# download source package and extract it to ~/pigsty, we just skip downloading here# curl https://github.com/pgsty/pigsty/releases/download/v3.6.0/pigsty-v3.6.0.tgz -o ~/pigsty.tgz; tar xzf ~/pigsty.tgz -C ~/scp ~/pigsty/dist/v3.6.0/pigsty-pkg-v3.6.0.el9.aarch64.tgz meta:/tmp/pkg.tgz
ssh meta
tar -xf pigsty.yml # extract pigsty source tarballcd pigsty # enter pigsty home dir./bootstrap # now bootstrap pigsty from local repo./configure # generate pigsty.ymlvi pigsty.yml # use local repo rather than install from the internet upstream repo#node_repo_modules: local./install.yml
Pigsty currently runs on nodes with Linux kernel and x86_64 / aarch64 architecture.
A “node” refers to a resource that is SSH accessible and offers a bare Linux OS environment.
It could be a physical machine, a virtual machine, or an OS-like container equipped with systemd, sudo and sshd.
It requires at least 1 node to deploy pigsty,
You can prepare more and setup everything in one-pass, or add them later.
The minimum node spec requirement is 1C1G, it is recommended to use at least 2C2G.
Higher is better, with no upper limit. Parameters will be automatically tuned based on available resource.
Use multiple nodes for production deployment
A functioning HA setup requires at least 3 nodes to work, or use 2 for a semi-HA setup
Spec
How many nodes do you need? It depends on your resource and your requirements.
Single Node Setup
The simplest setup with everything running on a single node, with four essential modules installed:
Pigsty will use /data as the default data directory, if you have a dedicated main data disk, it is recommended to mount it there,
and use /data1, /data2, /dataN for extra disk drivers.
Mount disks elsewhere?
If you are mounting it elsewhere, you’ll have to change the following parameters accordingly:
We recommend using ext4 or xfs as the filesystem for data disks. They have the best performance for PostgreSQL.
While ext4 has more tools for data recovery, xfs is more efficient for small files.
If you are running MinIO, It’s recommended to use xfs, otherwise, ext4 is recommended as the default option.
Network
Pigsty requires static network to work, you should explicitly assign a fixed IPv4 address for each node.
Don't have a fixed IP?
The 127.0.0.1 could be used as a workaround in case of no fixed IP address in one-node installation.
The IP address will be used as the node’s unique identifier, it should be the primary IP address bind to the primary network interface used for internal network communications.
Never use Public IP as identifier
Using public IP addresses as node identifiers can cause security and connectivity issues.
L2 VIP require L2 Networking
To use the optional Node VIP and PG VIP features, ensure all nodes are located within the same L2 network
Internet access is required when performing the standard (online) installation.
But pigsty can be offline installed via offline package,
which does not require Internet access in this case.
VIP
Pigsty supports optional L2 VIP for NODE clusters (keepalived) and PGSQL clusters (vip-manager).
To use L2 VIP features, You have to explicitly assign an L2 VIP for them.
It’s not a big deal when running on your own hardware,
but may become an issue when working in a public cloud environment.
Domain
Pigsty using local static domain names for the following service with WebUI.
You can assign your custom domain names to these services, or use real domain names.
Just change them in the infra_portal.
Domain names are optional, to use them, it is user’s responsibility to add the following records to your /etc/hosts file (local static resolution),
Or add them to your DNS server / public DNS vendor.
10.10.10.10 h.pigsty g.pigsty p.pigsty a.pigsty
2.2 - Linux OS
List of compatible linux OS Distributions
Recommended
Pigsty works well on EL9, U24, D12, Recommended OS: RockyLinux 9.6, Ubuntu 24.04.2, Debian 12.11
For extended support on legacy os, consider our pro subscription.
2.3 - Software
Locale, Firewall, Ansible, Pigsty…
Linux
Pigsty runs on Linux Operating System, it supports 14 mainstream Linux distros: Compatible OS List
We recommend using RockyLinux 9.6, Debian 12.11, or Ubuntu 24.04.5 as the default OS options.
Run pigsty on macOS?
You can install pigsty on macOS, and initiate control from your local laptop with ansible. (use as admin node)
But the database / infra / node / etcd services are still running on Linux nodes.
We strongly recommend using a clean installed OS environment with en_US set as the primary language.
How to enable en_US locale?
To ensure the en_US locale is available when using other primary language:
Pigsty DOES NOT use containers, main components are packed for specific distro major version.
Use the same OS version on all nodes
Please use the same major and minor version of the OS on all nodes in a single deployment.
File System
Pigsty suggests using ext4 or xfs file systems, both of which have the best performance for PostgreSQL use cases.
If you know what you are doing, you can also consider using file systems like zfs, but never run database services on network file systems like nfs.
If you need to use MinIO, it is recommended to use the xfs file system, which is the only file system recommended by MinIO.
It performs better in scenarios with a large number of small files, but the tool ecosystem (e.g., data recovery) is slightly inferior to ext4.
The default recommendation for running standard PostgreSQL services is to use the ext4 file system.
Firewall
Your security policy and firewall setup should allow access to the required ports.
To access the WebUI services, you’ll have to allow the HTTP (80) / HTTPS (443) access.
To access the PostgreSQL database services, you’ll have to allow the 5432 port for postgres.
You may access postgres services via other ports
5432: PostgreSQL database
6432: Pgbouncer Connection Pooler
5433: PG Primary Service
5434: PG Replica Service
5436: PG Default Service
5438: PG Offline Service
If you are accessing the postgres services via other ports, allow them accordingly
In a typical public cloud VPS setup, port 22/80/443/5432 are usually opened.
Expose database ports with caution
Exposing database service ports directly to the Internet is very dangerous.
If you need to do this, consider consulting Security Best Practices and proceed with caution.
In a typical production setup, port 22/80/443 are opened to DBA/OPS from LAN / Jumpserver.
And other ports are accessed from intranet. You have to make sure they are opened internally: used ports.
Ansible
Pigsty uses Ansible to initiate control from admin node to all managed nodes.
You don’t need to care about details, ansible is installed during the Bootstrap stage.
Install Ansible Manually
sudo apt install -y ansible python3-jmespath
sudo dnf install -y ansible python3-jmespath
sudo dnf install -y ansible python3.12-jmespath
sudo yum install -y ansible python-jmespath
brew install ansible
pip3 install jmespath
Ansible is only required on admin node, you can run ansible on macOS to use your laptop as admin node.
Pigsty
(RECOMMENDED) You can get & extract the latest stable version of pigsty source with:
The default main branch may in an unstable development status, git checkout v3.7.0 before using.
$ curl -fssL https://repo.pigsty.cc/get | bash -s v3.7.0
[v3.7.0]===========================================$ curl -fsSL https://repo.pigsty.cc/get | bash -s v3.7.0
[Docs] https://doc.pgsty.com
[Demo] https://g.pgsty.com
[Repo] https://github.com/pgsty/pigsty
[Download]===========================================[ OK ]version= v3.7.0 (from arg)curl -fSL https://repo.pigsty.cc/src/pigsty-v3.7.0.tgz -o /tmp/pigsty-v3.7.0.tgz
[WARN]tarball= /tmp/pigsty-v3.7.0.tgz exists, size= 1472486, use it
[ OK ]md5sums= df64ac0c2b5aab39dd29698a640daf2e /tmp/pigsty-v3.7.0.tgz
[Install]===========================================[ OK ]install= /home/vagrant/pigsty, from /tmp/pigsty-v3.7.0.tgz
[Bootstrap]===========================================[ OK ]ansible= ready
[ OK ]bootstrap= skip
you can run ./bootstrap to extrac offline package and install ansible
[TodoList]===========================================cd /home/vagrant/pigsty
./configure # [OPTIONAL] preflight-check and config generation./install.yml # deployment everything described by your config
You can also download the pigsty src (pigsty-<version>.tar.gz) from GitHub Release Page manually:
Pigsty requires an OS user with passwordless ssh and sudo on all managed nodes.
Naming Convention
Usually we’ll choose a name such as dba or admin for this purpose,
but avoid using root or postgres:
Avoid using root user
While possible, using root as admin user is not recommended for security reasons.
Don't use postgres dbsu as admin user
DBSU (postgres by default) should NOT be used as admin user.
It will cause unexpected security issues.
If you are using a different dbsu user, avoid using it as admin user as well.
Provide Password
The nopass requirement is optional if you can accept the password prompt for every ssh and sudo command.
Run playbook with password prompt
You can use the -k|--ask-pass when running playbook to prompt for the ssh password.
And use -K|--ask-become-pass to prompt for the sudo password.
./install.yml -k -K
Create Admin User
It’s user/vendor’s responsibility to create & deliver such an admin user during server provisioning stage.
But if you don’t have such an admin user, or that user is restricted, you can create one with pigsty itself:
Create admin user with pigsty
Assuming you have a root or existing admin user on the node, you can create an admin user with pigsty itself.
It will leverage the existing admin to create a new admin user.
It will create a dedicated dba (uid=88) user described by the following parameters,
with sudo / ssh properly configured.
All the admin user should have passwordless sudo privileges on all managed nodes.
In case you want to configure an admin user with passwordless sudo privileges from scratch:
Allow sudo without password
To manually allow a user to execute sudo commands without password:
Create a sudoers file for your admin user (assume vagrant, replace with your name choice):
echo'%vagrant ALL=(ALL) NOPASSWD: ALL'| sudo tee /etc/sudoers.d/vagrant
Assume your admin user name choice is dba, then /etc/sudoers.d/dba content would be
%dba ALL=(ALL) NOPASSWD: ALL
Ansible relies on sudo to execute commands with root privileges on managed nodes.
So on environments where sudo is not available (like inside a slim container), you may have to install sudo first.
SSH
Your current user should have nopass ssh access to all managed nodes as corresponding admin user.
Your current user can be the admin user itself, but not required as long as you can ssh as the admin user.
SSH configuration is Linux 101, but we will cover the basics here in case you are not familiar with it:
Pigsty will do that for you if you don’t have a key pair, during the bootstrap stage.
Copy SSH Key
You’ll need to distribute your generated public key remote (and local) servers, and put
it into the ~/.ssh/authorized_keys file of the admin user on all nodes.
The ssh-copy-id util can be used.
Distribute your ssh key to other nodes
Copy the public key to all managed nodes, ssh-copy-id or add to ~/.ssh/authorized_keys manually.
ssh-copy-id <ip> # Interactive password entry
You can use sshpass tool to pass the password directly without prompting, but it’s dangerous:
sshpass -p <password> ssh-copy-id <ip> # Non-interactive (use with caution)
Using Alias
When direct ssh access is not available (due to jumpserver, other port, credentials, etc…), consider:
Using SSH aliases
Configure SSH aliases in ~/.ssh/config, and put custom parameters for the alias there.
Host meta
HostName 10.10.10.10
User dba # <--- not the same user on remote IdentityFile /etc/dba/id_rsa # <--- not the ordinary key Port 24# <--- not the well-known port
And reference the alias in the inventory, use ansible_host to specify the real ssh alias.
nodes:hosts:# if node `10.10.10.10` requires an SSH alias `meta`10.10.10.10:{ansible_host:meta } # <---- access via `ssh meta`
Your should be able to access all managed nodes with nopass ssh via your current user from admin node.
And the remote user (admin user) should have the privilege to run nopass sudo commands.
Verify nopass ssh sudo is working
Run this command on admin node to all managed nodes:
ssh <ip|alias> 'sudo ls'
if there’s no password prompt or error raised, nopass ssh/sudo is working as expected.
2.5 - Sandbox
The 4-node env for learning & testing
Pigsty has a sandbox, which is a 4-node deployment with fixed IP addresses and other identifiers.
We will use this as a standard demonstration environment for learning and testing purposes.
Description
The sandbox consists of 4 nodes with fixed IP addresses and identities:
There’s a primary singleton PostgreSQL cluster: pg-meta on the meta node,
which can be used standalone, and there’s also an optional L2 VIP 10.10.10.2 and cluster DNS pg-meta bind to it.
10.10.10.10 meta pg-meta-1
10.10.10.2 pg-meta
There are three additional nodes in the sandbox, form a 3-instance PostgreSQL HA cluster pg-test.
With an optional L2 VIP 10.10.10.3 and cluster DNS pg-test bind to the cluster leader.
You are all set! use the make shortcuts to create VMs:
~/pigsty
make meta # 1-node devbox for quick start, dev, test & playgroundmake full # 4-node sandbox for HA-testing & feature demonstrationmake simu # 36-node simubox for production environment simulation...
make meta9 # create singleton-meta node with bento/rockylinux-9 imagemake full22 # create 4-node sandbox with generic/ubuntu2204 imagemake simu12 # create 36-node simulation env with generic/debian12 image
Configuration
You have to define VMs in the Vagrantfile before launch.
The default Vagrantfile defines a el9 (bento/rockylinux-9) 1-node virtual machine with the local virtualbox VM provider.
You can use specs with the config script, it will render the Vagrantfile according to spec and environment variables (resouce, image, vm provider, etc…).
cd ~/pigsty
vagrant/config [spec][image][scale][provider]vagrant/config meta # use the 1-node spec, default el8 imagevagrant/config dual el9 # use the 2-node spec, use el9 image insteadvagrant/config trio d12 2# use the 3-node spec, use debian12 image, double the cpu/mem resourcevagrant/config full u22 4# use the 4-node spec, use ubuntu22 image instead, use 4x cpu/mem resourcevagrant/config simu u24 1 libvirt # use the 36-node spec, use ubuntu24 image instead, use libvirt as provider instead of virtualbox
You can scale the resource unit with environment variable VM_SCALE, the default value is 1.
For example, VM_SCALE=2 vagrant/config meta will double the cpu / mem resources of the meta spec
You can create the VMs with vagrant up command after configuration.
Pigsty templates will use your ~/.ssh/id_rsa[.pub] as the default ssh key for vagrant provisioning.
Make sure you have a valid ssh key pair before you start, you can generate one by: ssh-keygen -t rsa -b 2048
There are some shortcuts that wrap the vagrant commands, you can use them to manage the VMs.
~/pigsty/vagrant
make # = make startmake new # destroy existing vm and create new onesmake ssh # write VM ssh config to ~/.ssh/ (required)make dns # write VM DNS records to /etc/hosts (optional)make start # launch VMs and write ssh config (up + ssh)make up # launch VMs with vagrant upmake halt # shutdown VMs (down,dw)make clean # destroy VMs (clean/del/destroy)make status # show VM status (st)make pause # pause VMs (suspend,pause)make resume # pause VMs (resume)make nuke # destroy all vm & volumes with virsh (if using libvirt)
Version
Pigsty currently uses the following vagrant boxes for testing:
It require extra setup to use the default 10.x.x.x CIDR as host-only networks
when using older version of virtualbox as vagrant provider: add it to /etc/vbox/networks.conf
echo"10.0.0.0/8"| sudo tee -a /etc/vbox/networks.conf
2.7 - Terraform
Provision cloud VM with terraform
Terraform is a popular IaC tool. You can create VMs on public cloud with one command.
Aliyun & AWS templates are used as example providers. You can take terraform.tf as an example.
Pigsty treats Infra & Database as Code. You can describe everything with declarative config Inventory.
Which is usually an Ansible Inventory in YAML format: pigsty.yml.
But CMDB can also be used as dynamic inventory.
The configure procedure will generate a config according to your environment and input.
But it’s OPTIONAL: you can always edit the pigsty.yml file directly as tutorial shows.
And there are plenty of templates for your reference.
Inventory
The main config file of Pigsty, Describe your entire deployment
Configure
Generate config file according to your input & environment
Tutorial
Planning your deployment according to business requirements
Template
Available configuration templates and examples
Security
Security considerations and best practices for prod deployment
CMDB
Use PostgreSQL as CMDB instead of local YAML config file
PGSQL
PostgreSQL Cluster with HA, PITR, IaC, ACL, Monitoring, Pooling
INFRA
Nginx, Repo, DNS, NTP, Prometheus and Grafana stack for Observability
NODE
Enroll nodes into the desired state and monitor it, and VIP, HAProxy
ETCD
Reliable distributed consensus storage (DCS), empowering PGSQL HA
Each pigsty deployment has a corresponding config inventory.
It could be stored in a local config file in YAML format, or dynamically generated from CMDB or any ansible compatible format.
Pigsty uses one monolith YAML config file by default, which is pigsty.yml, located in the pigsty home directory.
The configure script will generate the pigsty.yml file scaffold with good defaults according to your env and input,
but it’s OPTIONAL: you can always edit the pigsty.yml file directly as the tutorial shows.
Structure
The inventory consists of two parts: global vars & multiple groups. You can define new clusters in all.children.
And describe infra with global vars: all.vars. Which may look like this:
all:# Top-level object: allvars:{...} # Global Parameterschildren:# Group Definitionsinfra:# Group Definition: 'infra'hosts:{...} # Group Membership: 'infra'vars:{...} # Group Parameters: 'infra'etcd:{...} # Group Definition: 'etcd'pg-meta:{...} # Group Definition: 'pg-meta'pg-test:{...} # Group Definition: 'pg-test'redis-test:{...}# Group Definition: 'redis-test'# ...
There are plenty of examples under conf/, which can also be used as templates during configure.
Cluster
Each ansible group may represent a cluster, which could be a Node cluster, PostgreSQL cluster, Redis cluster, Etcd cluster, or Minio cluster, etc…
Cluster definition consists of two parts: hosts & vars.
You can define cluster members in <cls>.hosts and describe cluster with parameters in <cls>.vars.
Here’s an example of a 3-node HA PG cluster:
all:children:# All Groupspg-test:# Group Namehosts:# Group Host (Cluster Membership)10.10.10.11:{pg_seq:1, pg_role:primary }# Host110.10.10.12:{pg_seq:2, pg_role:replica }# Host210.10.10.13:{pg_seq:3, pg_role:offline }# Host3vars:# Group Vars (Cluster Parameters)pg_cluster:pg-test
The vars in cluster level will override the global vars, and vars in host level will override the cluster vars and global vars.
Parameter
Parameters are key-value pairs that define all entities in the deployment.
The key is a string name, and the value can be one of five types: boolean, string, number, array, or object.
And parameters can be set at different levels with the following precedence:
Level
Location
Description
Precedence
CLI Args
Command Line
via -e cli param arg
Highest (5)
Host Vars
<group>.hosts.<host>
Parameters specific to a single host
High (4)
Group Vars
<group>.vars
Parameters shared by hosts in a group/cluster
Medium (3)
Global Vars
all.vars
Parameters shared by all hosts
Low (2)
Defaults
<roles>/default/main.yml
Role implementation default values
Lowest (1)
Here are some examples about parameter precedence:
Override the PostgreSQL major version with Playbook CLI Args -e pg_version=16
Override pg instance role with Instance Level Parameter pg_role on Host Vars
Override pg cluster name with Cluster Level Parameter pg_cluster on Group Vars.
Specify global NTP servers with Global Parameter node_ntp_servers on Global Vars
If no pg_version is set, pigsty will use the default value from role implementation (18 by default)
Every parameter has a proper default value except for mandatory IDENTITY PARAMETERS; they are used as identifiers and must be set explicitly.
Such as pg_cluster, pg_role, and pg_seq in above snippet.
Available parameters vary according to the modules:
The configure script will generate the pigsty.yml config file inventory with good defaults according to your environment and input.
It’s OPTIONAL, you can edit the pigsty.yml directly as the tutorial shows.
Skip interactive wizard and using default/arg values
-n|--non-interactive
Non-interactively mode
-p|--port
Specify SSH port (only used if set)
Example
./configure # if there's only one IP address, otherwise it will ask you for it./configure -i 10.11.12.13 # just tell the primary IP address explicitly./configure -c rich -v 16# use the conf/rich.yml as template, with PG 16 and all extensions./configure -c app/supa # use the app/supa template, run self-hosting supabase./configure -c mssql # use the babelfish template, run MSSQL compatible PG kernel fork./configure -c full -s # use the 4-node sandbox config template, without IP replace & probe./configure -r china # use use china mirror instead of default repo
configure output example:
./configure
[vagrant@node-1 pigsty]$ ./configure
configure pigsty v3.7.0 begin
[ OK ]region= china
[ OK ]kernel= Linux
[ OK ]machine= x86_64
[ OK ]package= rpm,dnf
[ OK ]vendor= rocky (Rocky Linux)[ OK ]version=9(9.5)[ OK ]sudo= vagrant ok
[ OK ]ssh=[email protected] ok
[WARN] Multiple IP address candidates found:
(1) 192.168.121.193 inet 192.168.121.193/24 brd 192.168.121.255 scope global dynamic noprefixroute eth0
(2) 10.10.10.11 inet 10.10.10.11/24 brd 10.10.10.255 scope global noprefixroute eth1
[ IN ] INPUT primary_ip address (of current meta node, e.g 10.10.10.10):
=> 10.10.10.11
[ OK ]primary_ip= 10.10.10.11 (from input)[ OK ]admin=[email protected] ok
[ OK ]mode= meta (el9)[ OK ]locale= C.UTF-8
[ OK ] configure pigsty doneproceed with ./install.yml
Behavior
Config Template
If -c|--conf <template> is specified, it will generate the config file from the specified template. Such as meta, app/supa, etc…
If no config template is given, it will use the default 1-node config template meta.
IP Address
If -i|--ip <ipaddr> is specified, it will replace the placeholder 10.10.10.10 in the config template with the given IP address.
Otherwise, if the current node has only one IP address, it will be used. If there are multiple IP addresses, it will ask you to input the primary IP address of current node manually.
PostgreSQL Version
If -v|--version is specified, it will use the specified PostgreSQL major version number, range from 13 to 18.
If no version is specified, it leaves the pg_version intact, which usually fallbacks to 18 by default.
Region
If -r|--region is specified, it will use the specified region directly. The china mirror will be used where Google services are not accessible.
Proxy Environment
If -x|--proxy is specified, it will write the current proxy environment variables to the config proxy_env.
Will be reused during installation. Including: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, NO_PROXY.
Skip Mode
If -s|--skip is specified, it will skip IP address replacement and ssh sudo privilege check
Non-Interactive Mode
If -n|--non-interactive is specified, this script will not ask you for anything, but you have to specify a primary IP address with -i|--ip <ipaddr> explicitly.
SSH Port
If -p|--port is specified, it will use the specified SSH port instead of the default 22.
It is used when your local SSH port is not 22.
Low-End Hardware Optimization
If current node CPU cores ≤ 4, it will use the tiny mode for pg_conf and node_tune to optimize for low-end hardware.
Locale Setting
Pigsty will use C.UTF-8 as the default locale, If:
PostgreSQL major version ≥ 17, with built-in local provider (default)
OR, Your system supports C.utf8 / C.utf-8 locale (locale -a)
Otherwise, local C will be used by default.
3.3 - Tutorial
Forge a complex config from scratch
Instead of using configure to generate config, you can write a pigsty config file from scratch manually.
Here is a tutorial to help you to build a complex config file inventory from scratch.
Minimal
This is a minimal working config example, you have to tell pigsty the IP of admin node and infra node.
We add two new groups here: etcd and pg-meta, which defines an 1-node ETCD cluster and a 1-node PGSQL cluster.
Recreate everything with ./install.yml, or using these cmd to do incremental steps:
~/pigsty
./etcd.yml -l etcd # install etcd module on group etcd./pgsql.yml -l pg-meta # install pgsql module on group pg-meta
The PGSQL module depends on ETCD for HA consensus, so make sure to install ETCD module first.
Database & Users
Now we want to customize our postgres database cluster, with users, database, and backups:
We define some additional details at the pg-meta cluster level:
pg_users: define a new user dbuser_meta with password DBUser.Meta
pg_databases: define a new database meta with pigsty CMDB schema and vector extension
node_crontab: define crontab that takes a full backup at 1 am every day
Instead of recreating everything with ./install.yml, we make changes incrementally:
~/pigsty
bin/pgsql-user pg-meta dbuser_meta # create user dbuser_meta on pg-metabin/pgsql-db pg-meta meta # create database meta on pg-meta./node.yml -l pg-meta -t node_crontab # add backup task to crontab on pg-meta
PG VER & Extensions
You can install different major versions of PostgreSQL, and 437+ corresponding extensions.
Let’s install PostgreSQL 16 (instead of default 18) with timescaledb, postgis, and pgvector extensions.
~/pigsty/pigsty.yml
all:children:infra:{hosts:{10.10.10.10:{infra_seq:1}}}etcd:{hosts:{10.10.10.10:{etcd_seq:1 } }, vars:{etcd_cluster:etcd } }pg-meta:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-metapg_users:- name:dbuser_metapassword:DBUser.Metapgbouncer:trueroles:[dbrole_admin]comment:pigsty admin userpg_databases:- name:metabaseline:cmdb.sqlcomment:pigsty meta databaseschemas:[pigsty]extensions:[vector, postgis, timescaledb ] # <--- CREATE EXTENSIONSpg_libs:'timescaledb, pg_stat_statements, auto_explain'# <--- LOAD EXTENSIONSnode_crontab:- '00 01 * * * postgres /pg/bin/pg-backup full'vars:admin_ip:10.10.10.10region:default # use local mirror for faster download speed # <--- default|china|europerepo_extra_packages:[timescaledb, postgis, pgvector, pgsql ]# <--- DOWNLOAD EXTENSIONSpg_extensions:[timescaledb, postgis, pgvector ] # <--- INSTALL EXTENSIONSpg_version:16# PG 17 is the default latest major version # <--- USE PG 16 VERSION
repo_extra_packages: download timescaledb and postgis extensions.
Let’s redownload missing packages (PG 16 kernel & extensions), remove the old cluster, and recreate it:
make repo # redownload packages./pgsql-rm.yml -l pg-meta # remove the old pg-meta cluster (because it is PG18)./pgsql.yml -l pg-meta # recreate the pg-meta cluster with PG16 & extensions
Pigsty’s working assumption is there’s only ONE postgres instance on each node.
There’s no support to run multiple postgres instances on a single node.
Redis Up
Pigsty has optional Redis support, which is used as a cache in front of PostgreSQL.
This directory conf contains pigsty config templates, which will be used during configure procedure.
Config templates can be designated using ./configure -c <conf>, where the conf is a relative path to conf directory (with or without .yml suffix).
e.g. the ~/pigsty/conf/rich.yml can be specified as rich
./configure # use the meta.yml config template by default./configure -c meta # use the meta.yml 1-node template explicitly./configure -c rich # use the 1-node template with all extensions & minio./configure -c slim # use the minimal 1-node template./configure -c supabase # use the Supabase 1-node template./configure -c app/dify # use the dify app template
If no -c|--conf is given, the single-node meta config template will be automatically selected by default.
Basic Templates
Here are one-node templates, which provide different features and configurations.
Trade off between availability and consistency for PostgreSQL
pg_rpo : trade-off between Availability and Consistency
pg_rto : trade-off between failure chance and impact
Access
Use VIP, DNS, HAProxy instead of fixed IP
Do not access the database directly via a fixed IP address; use VIP, DNS, HAProxy, or their combination.
Haproxy will handle the traffic control for the clients in case of failover/switchover.
3.6 - CMDB
Use PostgreSQL as config inventory
Pigsty allows you to use a database (CMDB) as a dynamic configuration source instead of a static configuration file.
You can use the built-in PostgreSQL as config inventory for config management.
With Postgres CMDB, configuration is organized in structured relational tables, which can be easily queried and manipulated using SQL.
This allows for easier integration with other systems and tools.
How does it work?
Ansible allows you to use a dynamic inventory script to generate the inventory config on-the-fly.
The idea is to replace static pigsty.yml in ansible.cfg with a dynamic shell script inventory.sh
The CMDB baseline schema is shipped with pigsty: files/cmdb.sql
And most of the default config templates will use it as example baseline. Which means it can be used by default.
all:children:pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_databases:- name:metabaseline:cmdb.sql # <--- use this as database schema baseline
Load Config Data
CMDB is empty by default, load config file into the CMDB with the bin/inventory_load script.
Run bin/inventory_load without arguments will load the default pigsty.yml into the default CMDB.
usage: inventory_load [-h][-p PATH][-d CMDB_URL]load config arguments
optional arguments:
-h, --help show this help message and exit -p PATH, --path PATH config path, ${PIGSTY_HOME}/pigsty.yml by default
-d DATA, --data DATA postgres cmdb pgurl, ${METADB_URL} by default
Use -p to specify the config file path, and -d to specify the CMDB URL.
High-performance in-memory cache, optional data structure server
4.1 - Ansible
Get started with basic ansible concepts
Pigsty implements admin controllers with Ansible, which is an open source automation tool for managing large-scale infrastructure in an Infra-as-Code (IaC) manner.
Widely used in the industry by operators.
Install
Pigsty will try its best to install ansible and its dependencies during bootstrap.
But you can always install it manually, it is available on most OS’s official repos and can be installed with one command.
Install Ansible
Playbooks also require a weak dependency: the jmespath python package.
cd ~/pigsty; ./bootstrap
sudo apt install -y ansible python3-jmespath
sudo dnf install -y ansible python-jmespath
sudo dnf install -y ansible python3.12-jmespath
sudo yum install -y ansible python-jmespath
brew install ansible
pip3 install jmespath
Beware that el10 epel does not offer the ansible package, which is fixed by Pigsty PGDG el10 repo
macOS
Ansible is available on macOS too. You can install Ansible on your Mac with Homebrew.
And use it as the admin node to manage remote cloud server.
It’s convenient if you are deploying a single-node pigsty on cloud VPS. But not recommended for production use.
Basics
Knowledge about Ansible is good but NOT REQUIRED. You only need to know how to run Ansible Playbooks.
Playbooks are executable YAML files that contain a series of tasks to be executed.
Running the ./node.yml playbook essentially translates to ansible-playbook node.yml.
The hashbang at the top of the file makes it directly executable.
And you can use Args to control the playbook execution:
~/pigsty
./node.yml # run infra playbook on all nodes./pgsql.yml -l pg-test # run pgsql playbook on pg-test cluster./infra.yml -t repo # run subtask repo of infra.yml./pgsql-rm.yml -e pg_rm_pkg=false# remove pgsql, but keep packages
The following 4 parameters need your attention to use ansible effectively:
The execution target of a playbook can be limited with -l|--limit <selector>.
It is handy when trying to run playbooks on a specific host/node or group/clusters.
Here are some examples of host limits:
./pgsql.yml # run on all hosts (dangerous!)./pgsql.yml -l pg-test # run on pg-test cluster./pgsql.yml -l 10.10.10.10 # run on single host 10.10.10.10./pgsql.yml -l pg-* # run on host/group matching glob pattern `pg-*`./pgsql.yml -l '10.10.10.11,&pg-test'# run on 10.10.10.11 of group pg-test./pgsql-rm.yml -l 'pg-test,!10.10.10.11'# run on pg-test, except 10.10.10.11./pgsql.yml -l pg-test # Execute the pgsql playbook against the hosts in the pg-test cluster
Running playbook without host limit can be Dangerous!
Missing this value could be dangerous, since most playbooks will execute on all hosts. DO USE WITH CAUTION.
Limit Task
The execution tasks can be controlled with -t|--tags <tags>.
If specified, tasks with given tags will be executed instead of the ENTIRE playbook.
Here are some task limit examples:
To run multiple tasks, specify multiple tags and separate with comma: -t tag1,tag2:
./node.yml -t node_repo,node_pkg # add repo, then install packages./pgsql.yml -t pg_hba,pg_reload # config, then reload pg hba rules
Extra Vars
You can override config param at runtime with cli args, it has the highest precedence.
Extra command-line args can be passed via -e|--extra-vars KEY=VALUE, it can be used multiple times:
# create admin with another admin user./node.yml -e ansible_user=admin -k -K -t node_admin
# init a specific redis instance: 10.10.10.11:6379./redis.yml -l 10.10.10.10 -e redis_port=6379 -t redis
# remove postgres, but keeps packages and data./pgsql-rm.yml -e pg_rm_pkg=false -e pg_rm_data=false
for complex parameters, JSON string can be used:
# add repo and install package./node.yml -t node_install -e '{"node_repo_modules":"infra","node_packages":["duckdb"]}'
Designate Inventory
The default config file is pigsty.yml in the pigsty home directories.
You can use the -i <path> parameter to specify a different Inventory file path.
./pgsql.yml -i conf/rich.yml # initialize a single node with all extensions downloaded according to rich config./pgsql.yml -i conf/ha/full.yml # initialize a 4-node cluster according to full config./pgsql.yml -i conf/app/supa.yml # initialize a 1-node Supabase deployment according to supa.yml config
Change Default Inventory File
To permanently change the default config file, change the inventory parameter in the ansible.cfg.
4.2 - Playbook
Run playbooks with ansible
Pigsty implements admin controllers with idempotent Ansible playbooks.
Playbooks require the ansible-playbook executable bin in your PATH. You’ll have to install ansible to run playbooks.
Here are built-in playbooks in Pigsty, you can also add your own.
The special playbook install.yml will deploy everything with ad hoc playbooks:
Playbook
Command
Group
infra
[nodes]
etcd
minio
[pgsql]
infra.yml
./infra.yml
-l infra
✓
✓
node.yml
./node.yml
✓
✓
✓
✓
etcd.yml
./etcd.yml
-l etcd
✓
minio.yml
./minio.yml
-l minio
✓
pgsql.yml
./pgsql.yml
✓
Circular Dependency Between NODE and INFRA
There is a weak circular dependency between NODE and INFRA: to register a NODE
to INFRA, the INFRA should already exist, while the INFRA module relies on NODE to work.
One way to work around it is to init infra nodes first, then add other nodes, if you wish to deploy them all in
one-pass, install.yml is the way to go.
Safety Considerations
Idempotent but Potentially Destructive
Most playbooks are idempotent, meaning that some deployment playbooks may erase existing databases and create
new ones without the protection option turned on. Especially care with pgsql, minio, and infra playbooks.
Please read the documentation carefully and operate with caution.
The author is not responsible for any loss of databases due to misuse.
Safety Best Practices
Read playbook documentation carefully before execution
Ctrl-C to stop immediately if you see something wrong
Start with non-production environments for testing
Limit execution hosts (-l) to avoid unintended hosts if applicable
Use specific tags (-t) to run subset of tasks if possible
Dry Run Mode
# Preview what would be changed without actually executing./pgsql.yml -l pg-test --check --diff
# Combine with tags to check specific tasks./pgsql.yml -l pg-test -t pg_config --check --diff
4.3 - Nginx Portal
Configure infra portal and nginx settings
Pigsty installs Nginx on the INFRA Node as a web service proxy, using ports 80/443 by default.
The global parameter infra_portal configures Nginx proxy rules and upstream services.
The Nginx server configuration is specified through the infra_portal parameter.
Users declare all domains to be proxied through Nginx, along with corresponding upstream server endpoints or local directory paths.
Use these playbook tasks to manage local package repositories (YUM/APT) on Infra nodes:
./infra.yml -t repo # Create local repo from internet or offline packages./infra.yml -t repo_dir # Create local repo directory./infra.yml -t repo_check # Check if local repo exists./infra.yml -t repo_prepare # Use existing local repo if available./infra.yml -t repo_build # Build local repo from upstream if not exists./infra.yml -t repo_upstream # Add upstream repo/list files./infra.yml -t repo_remove # Remove existing repo files if repo_remove=true./infra.yml -t repo_add # Add upstream repo files to /etc/yum.repos.d (or apt)./infra.yml -t repo_url_pkg # Download packages defined in repo_url_packages./infra.yml -t repo_cache # Create metadata cache with yum makecache / apt update./infra.yml -t repo_boot_pkg # Install bootstrap packages (createrepo_c, yum-utils, etc)./infra.yml -t repo_pkg # Download packages & deps from upstream./infra.yml -t repo_create # Create local repo with createrepo_c / dpkg-dev./infra.yml -t repo_use # Add new repo to /etc/yum.repos.d | apt sources./infra.yml -t repo_nginx # Start nginx as file server if not running
Commonly used commands:
./infra.yml -t repo_upstream # Add upstream repos defined in repo_upstream./infra.yml -t repo_pkg # Download packages and their dependencies./infra.yml -t repo_create # Create/update local yum/apt repo
4.5 - DNS Domain
Setup domain names for Web Services
After installing Pigsty, users can access most Infra components’ web interfaces via IP + Port.
Let’s say your node’s internal IP is 10.10.10.10, then by default:
While IP + Port works fine for dev/test environments (hey, we’re all lazy sometimes!), for more serious deployments, I strongly recommend accessing these services via domain names.
Using domains has numerous advantages, doesn’t cost extra, and requires just one simple config line.
MacOS: Double-click ca.crt, add to Keychain, search pigsty-ca, open and “Trust” the root cert.
Windows: Add ca.crt to “Trusted Root Certification Authorities”.
After trusting Pigsty’s CA, no more “untrusted certificate” warnings! 🎉
Public Domain Resolution
Use DNS providers like Cloudflare, Godaddy, Aliyun, or Tencent Cloud DNSPod.
Requires purchasing a domain - basic ones cost ~$10/year.
Add DNS records via provider’s console/API to point domains to Pigsty’s public IP.
Example: With domain pigsty.xxx, add wildcard * A record or individual A records:
h.pigsty.xxx → 1.2.3.4
a.pigsty.xxx → 1.2.3.4
p.pigsty.xxx → 1.2.3.4
g.pigsty.xxx → 1.2.3.4
Pigsty includes Certbot support for free HTTPS certs (renew every 3 months).
Further Reading
For more advanced configurations, check the Pigsty documentation for DNS, Nginx, and HTTPS certificate management.
4.6 - SSL Certs
Configure real & self-signed HTTPS certs
Pigsty comes with Certbot pre-installed on the Infra node, enabling you to obtain free Let’s Encrypt HTTPS certificates for Nginx servers and public domains.
Prerequisites
Before obtaining Let’s Encrypt certificates, ensure you have:
A public domain name
DNS records pointing to your server’s public IP
Nginx properly configured with your domains
Step 1: Determine Which Domains Need Certificates
First, identify which upstream services require public certificates by configuring domains in your infra_portal:
I’m Ruohang Feng, also known as @Vonng, creator of Pigsty.
I’ve developed most of Pigsty’s code solo,
with the community contributing specific features.
Unique individuals create unique works — I hope Pigsty can be one of those creations.
If you are interested in the author, here’s my personal website: https://vonng.com/en/
Origin Story
The Pigsty project kicked off between 2018 and 2019, originating from Tantan,
a dating app similar to China’s Tinder, now acquired by Momo.
Tantan, a startup with a Nordic vibe, was founded by a team of Swedish engineers.
Renowned for their tech sophistication, they chose PostgreSQL and Go as their core tech stack.
Tantan’s architecture, inspired by Instagram, revolves around PostgreSQL.
They managed to scale to millions of daily active users, millions of TPS, and hundreds of TBs of data using PostgreSQL exclusively.
Almost all business logic was implemented using PG stored procedures, including recommendation algorithms with 100ms latency!
This unconventional development approach, deeply leveraging PostgreSQL features,
demanded exceptional engineering and DBA skills. Pigsty emerged from these real-world,
high-standard database cluster scenarios as an open-source project encapsulating our top-tier PostgreSQL expertise and best practices.
Dev Journey
Initially, Pigsty didn’t have the vision, objectives, or scope it has today.
It was meant to be a PostgreSQL monitoring system for our use.
After evaluating every available option—open-source, commercial, cloud-based, datadog, pgwatch,……
none met our observability bar. So, we took matters into our own hands, creating a system based on Grafana and Prometheus,
which became the precursor to Pigsty. As a monitoring system, it was remarkably effective, solving countless management issues.
Eventually, developers wanted the same monitoring capabilities on their local dev machines.
We used Ansible to write provisioning scripts, transitioning from a one-off setup to a reusable software.
New features allowed users to quickly set up local DevBoxes or production servers with Vagrant and Terraform,
automating PostgreSQL and monitoring system deployment through Infra as Code.
We then redesigned the production PostgreSQL architecture, introducing Patroni and pgBackRest
for high availability and point-in-time recovery.
We developed a zero-downtime migration strategy based on logical replication,
performing rolling updates across 200 database clusters to the latest major version using blue-green deployments.
These capabilities were integrated into Pigsty.
Pigsty, built for our use, reflects our understanding of our needs, avoiding shortcuts.
The greatest benefit of “eating our own dog food” is being both developers and users,
deeply understanding and not compromising on our requirements.
We tackled one problem after another, incorporating solutions into Pigsty.
Its role evolved from a monitoring system to a ready-to-use PostgreSQL distribution.
At this stage, we decided to open-source Pigsty, initiating a series of technical talks and promotions,
attracting feedback from users across various industries.
Full-time Startup
In 2022, Pigsty secured seed funding from Dr. Lu Qi’s MiraclePlus S22 (Former YC China),
enabling me to work on it full-time. As an open-source project, Pigsty has thrived.
In the two years since going full-time, its GitHub stars skyrocketed from a few hundred to 4000,
On OSSRank, Pigsty ranks 26th among PostgreSQL ecosystem projects.
Originally only compatible with CentOS7, Pigsty now supports all major Linux Distros and PostgreSQL versions 12 - 17, integrating over 420 extensions from the ecosystem. I’ve personally compiled, packaged, and maintained some extensions not found in official PGDG repositories.
Pigsty’s identity has evolved from a PostgreSQL distribution to an open-source cloud database alternative, directly competing with entire cloud database services offered by cloud providers.
Cloud Rebel
Public cloud vendors like AWS, Azure, GCP, and Aliyun offer many conveniences
to startups but are proprietary and lock users into high-cost infra rentals.
We believe that top-notch database services should be as accessible as the
top-notch database kernel (PostgreSQL), not confined to costly rentals from cloud providers.
Cloud agility and elasticity are great, but it should be open-source,
local-first and cheap enough. We envision a cloud computing universe with an open-source solution,
returning the control to users without sacrificing the benefits of the cloud.
Thus, we’re leading the “cloud-exit” movement in China,
rebelling against public cloud norms to reshape industry values.
Our Vision
We’d like to see a world where everyone has the factual right to use top services freely,
not just view the world from the pens provided by a few public cloud providers.
This is what Pigsty aims to achieve —— a superior, open-source, free RDS alternative.
Enabling users to deploy a database service better than cloud RDS with just one click, anywhere (including on cloud servers).
Pigsty is a comprehensive enhancement for PostgreSQL and spicy satire on cloud RDS. We offer “the Simple Data Stack”,
which consists of PostgreSQL, Redis, MinIO, and more optional modules.
Pigsty is entirely open-source and free, sustained through consulting.
A well-built system might run for years without issues, but when database problems arise, they’re serious.
Often, expert advice can turn a dire situation around, and we offer such services to clients in need—a fairer and more rational model.
5.2 - License
Open Source license and BOM Inventory
Pigsty uses the AGPLv3 license, which is a strong copyleft license that requires you to also distribute the source code of your derivative works under the same license when you “Distribute” Pigsty.
It’s no big deal if you are just “USING” it.
Permissions
Commercial use
- Modification
- Distribution
- Patent use
- Private use
<div class="col-span-5 h-full">
Conditions
License and copyright notice
- State changes
- Disclose source
- Network use is distribution
- Same license
<div class="col-span-3 h-full">
Limitations
Liability
- Warranty
Exemptions
We extend exemptions to common end users under terms akin to the Apache 2.0 license. As long as you do not use pigsty for DBaaS and OEM purpose (i.e. re-sell as database service).
Our Subscription includes coverage for DBaaS & OEM scenarios.
Why AGPLv3
We don’t like the idea that public cloud vendors take open-source code, provide as-a-service, and not give back equally (their admin controller & console) to the community. This is a vulnerability in the GPL license that AGPLv3 was designed to close.
The AGPLv3 does not affect regular end users: using Pigsty internally is not “distributing” it, so you don’t have to worry about whether your business code needs to be open-sourced. If you do worry about it, you can always choose the pro version with written guarantees.
While you only need to consider AGPLv3 when you “distribute” Pigsty or modifications to it as part of a software/service offering. Such as database/software/cloud vendors who provide Pigsty as a service or part of their software to their customers.
For 437 PostgreSQL Extensions License, check the extension list for details
Necessity Levels:
Required: Essential core components, no option to disable.
Recommended: Enabled by default, can be disabled.
Optional: Not enabled by default, can be installed.
On Demand: Peripheral tools that are completely optional.
Content
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
Copyright (C) 2018-2025 Ruohang Feng, Author of Pigsty
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.
5.3 - Community
Open source community and user groups
The Pigsty community already offers free WeChat/Discord/Telegram Q&A Office Hours, and we are also happy to provide more free value-added services to our supporters.
GitHub HQ
Consider giving us a star on GitHub if you find Pigsty useful.
Issues, PRs, and discussions are welcome.
Repo
The source code repository of Pigsty
Organization
The `pgsty` organization on GitHub is the official home of Pigsty.
Issues
Create new issues and report bugs, submit feature requests
Discuss
Ask questions, share ideas, and get help from the community
Community
We have 7 active wechat user groups with around 2700+ users, join our discussions!
If you are using a cloud provider, please tell us which cloud provider and what operating system image you are using.
If you have customized and modified the environment after installing the bare OS, or have specific security rules and firewall configurations in your WAN, please also tell us when troubleshooting.
Pigsty Config File (REQUIRED)
Don’t forget to remove sensitive information like passwords, etc…
cat ~/pigsty/pigsty.yml
What did you expect to happen?
Please describe what you expected to happen.
How to reproduce it?
Please tell us as much detail as possible about how to reproduce the problem.
Monitoring Screenshots
If you are using pigsty monitoring system, you can paste RELEVANT screenshots here.
Error Log
Please copy and paste any RELEVANT log output. Do not paste something like “Failed to start xxx service”
Syslog: /var/log/messages (RHEL) or /var/log/syslog (Debian)
Lighting Talk: Why PostgreSQL is the king of DB in the AI area.
Lighting Talk: Trapped! Best practice for delivering PostgreSQL!
The 8th Conf of PG Ecosystem, HangZhou, China
2025-05-16
Extension Delivery, Make your PGEXT accessible to users
Giving a lightning talk here at #PGConfdev is Ruohang Feng, founder at Pigsty, presenting on “Extension Delivery: Make your PG Ext accessible to users” —— PGConf.Dev X
Scope: Pigsty v2.0.0 - v2.5.1, fixed in Pigsty v2.6.0
Description:
etcd has a default 2GB database capacity limit. If your etcd database exceeds this limit, etcd will reject write requests, which may cause PostgreSQL high availability mechanisms that depend on etcd to fail. At the same time, etcd’s data model creates a new version with each write, so even if you have only a few keys, frequent writes to your etcd cluster can cause the etcd database size to continuously grow and eventually fail when it reaches the capacity limit.
Solution:
Update Pigsty to v2.6.0 or above, or update the code in the roles/etcd section and re-execute ./etcd.yml to force reset the etcd cluster to implement the fix.
Pigsty follows a structured development roadmap with regular releases and continuous improvement. This page outlines our release schedule, upcoming features, and long-term plans.
Release candidates: v3.1.0-rc1 (production ready, final testing)
Release Recommendations
Always use tagged releases rather than the GitHub main branch
Use version-specific releases for production deployments
Test new versions in development environments before upgrading production
Feature Radar
The following features are planned or under consideration for future releases:
PostgreSQL 18 support
Tutorial: Supabase Self-Hosting
Tutorial: Dify Self-Hosting
Tutorial: Odoo Self-Hosting
EL 10 support
Self-hosting PostHog
Monitoring deployments of MySQL databases
Replace promtail with vector
Replace loki with victorialogs
Replace prometheus with victoriametrics
Better log sink destination management
Deploying and monitoring high-availability Kubernetes clusters with SealOS!
6 - Values Proposition
Pigsty’s 8 core value propositions that deliver infinite possibilities
Pigsty delivers eight core value propositions, giving you an enterprise-ready database platform
Extensible Postgres
Blossom of Possibilities
Reliable Infra
Rock-Solid and Secure
Observable Graphics
Clarity and Vision
Scalable Service
Elastic Performance
Maintainable Toolbox
Simple and Actionable
Composable Modules
Flexible Lego-Blocks
Controllable FOSS
Sovereign Self-Hosting
Affordable Solution
Cost-Effective RDS
🧩 Extensible Postgres
“Nurturing all, thriving in synergy, forging infinite possibilities!”
Analytics
Big Data's New Challenger
AI Ready
Baseline for RAG/Vector App
Geospatial
De Facto GIS Standard
Time Series
Temporal Data Mastery
Text Search
Built-in Search Engine
Languages
Language of Your Choice
FDW Federation
Connecting Data Silos
Featured
Database as a Platform
🛡️ Reliable Infra
“Towering peaks, bedrock solid, standing firm at any summit!”
High-Availability
Reliable PostgreSQL RDS
Self-Healing
Adaptive Service Failover
PITR Protection
Pre-configured Backup & Archives
Infra Closure
No External Dependencies
Access Control
Built-in Best-Practice Model
Confidentiality
Guaranteed Data Security
Data Integrity
Thorough Verification
Battle-Tested
Availability Results
📊 Observable Graphics
“Heaven’s movement, all-seeing view, perceiving details to master the whole!”
Monitoring Infra
Built-in Observability Stack
Data-Driven
Measure what you Manage
SOTA Experience
The Definitive PG monitoring
Universal Monitoring
RDS or Compat Kernels
Automatic Alerts
No more manual checks
Performance Tuning
Slow-query Optimize
Log Analysis
Fast root-cause detection
Custom Dashboards
Low-code Data App
⚡ Scalable Service
“Ever-flowing like water, soft yet resilient, converging streams to adapt to endless change!”
Great Performance
Hardware Fully Harnessed
R/W Separation
Unlimited Read Scaling
Connection Pooling
High Concurrency
Load Balancing
Traffic Control
Horizontal Scaling
Distributive Extension
Storage Expansion
Transparent Compression
Mass Deployment
Large clusters made easy
Elasticity
Cloud-like Elasticity
🔧 Maintainable Toolbox
“Blazing like wildfire, illuminating all around, burning bright without end!”
Infra as Code
Define everything in Code
Simple & Easy
Up and Running in Minutes
Bare Linux
No Containers or Kubernetes
Offline Install
Stable, Hassle-free Delivery
Admin SOP
Best Practices Included
No Downtime
Online Migration & Resizing
Rich Parameters
Plenty of Tunable Knobs
Provisioning
One-command IaaS Provisioning
🎯 Composable Modules
“Swift as the wind, simplifying complexity, riding the currents of change with freedom and ease!”
Modular Design
Lego-like Assembly
App Templates
One-Click Enterprise Deployment
Core Modules
Fully-Featured Postgres RDS
Extra Modules
Extending RDS Capabilities
Kernel Modules
Swappable Database Engines
OLAP Modules
Powerful Analytics Capabilities
Pilot Modules
Exploring Cutting-Edge Frontiers
Flavor Modules
Creative Postgres Flavors
🎛️ Controllable FOSS
“Grounded like the earth, gathering all rivers—standing firm while gazing at the stars!”
Software Freedom
Self-hosting democratized
Local-First
Run on-premises indefinitely
Multi-Cloud
No vendor lock-in
Free Extensions
437 PostgreSQL extensions
Data Ownership
Pay fair cost for resources
Friendly License
AGPLv3 preserving freedom
Compliance Ready
Meeting domestic requirements
Expert Support
Top-tier PostgreSQL specialists
💰 Affordable Solution
“Thunderous impact, breaking to build anew, keeping costs manageable and value ever rising!”
Open-Source
Fully leverage PostgreSQL's eco
Save More
Escape the RDS money pit
DBA Efficiency
Everyone can be a DBA
Simplified Arch
No Containers or K8s
Enable Cloud-Exit
Key blockers resolved
Community Support
Discuss & Share
Expert Consultation
Pay as needed
Subscription
Clear pricing, value for money
These eight values work in synergy to deliver a comprehensive database platform that scales from development to enterprise production environments. Pigsty transforms PostgreSQL from a simple database into a powerful, observable, and maintainable data infrastructure that organizations can truly own and control.
6.1 - Extensible Postgres
Blossom of Possibilities
Nurturing all, thriving in synergy, forging infinite possibilities!
pgrouting: GIS spatial path planning, and graph computation
pointcloud: Storage and processing of LiDAR point cloud data
Time Series: Temporal Data Mastery
Enhanced time-series processing & analytics capabilities, for IoT, FinTech, APM
Simplify app dev with temporal tables, time travel, version control, cron tasks
timescaledb: Process & Analysis Timeseries, continuous agg of streaming event
temporal_tables/e-maj: Time travel capabilities
table_version: Version control for table data
pg_cron/pg_task: Background task management and scheduling
Text Search: Built-in Search Engine
Built-in full-text search engine with multilingual support
Advanced text processing and search capabilities
Built-in full-text search with tsvector and tsquery
Multilingual support with various dictionaries
Fuzzy matching and similarity search
Advanced text analytics and processing
Languages: Language of Your Choice
PostgreSQL is not just a database, but a full-featured data development platform
Support for 20+ stored procedure languages, putting even Oracle to shame
Multiple Languages: Python, Js, Java, Shell, Perl, Lua, TCL, PRQL, R, and more
pg_tle: Framework for developing extensions using trusted languages
Stored Procedure Toolkit: Debugging, Profiling, Validating, Unit testing
FDW Federation: Connecting Data Silos
Create data federations with FDW, becoming the data exchange hub
Uniformly access & ETL all heterogeneous data sources with SQL
Major Relational Databases: pgsql, mysql, mssql, oracle, db2, duckdb, sqlite
Various Data Sources: redis, kafka, mongo, hdfs, s3, log, jdbc, odbc
20+ FDWs available, write FDWs using WASM
Multicorn: Developing FDWs using Python
Featured: Database as a Platform
HTTP requests, SMTP email, compression, QR codes, PDF scanning
Developing web app inside Postgres - Just use PostgreSQL for everything!
Self-healing architecture with transparent topology and multi-writer experience.
Automatic traffic routing based on health checks and flexible access methods.
Industry HA best practices: Patroni, Etcd, HAProxy, VIP Manager
Self-signed CA, SSL encryption, password for backups and endpoints
As long as your passwords & key are secure, your data remains safe.
Local CA-issued certificates with SSL/TLS enabled globally by default
SCRAM-SHA-256 verification, AES-encrypted backup data
Precise allow/deny list default policies to prevent unauthorized access
All externally exposed web services are uniformly managed through Nginx
Data Integrity: Thorough Verification
Data checksums for silent corruption. Replicas and delayed standby instances.
Audit extensions and centralized logging, with transparency and efficiency.
CRIT configuration template: optimized for data integrity
Data checksums enabled to prevent silent data corruption
WatchDog: STONITH to prevent cluster split-brain
Audit Log: centrally collected, tamper-proof
Battle-Tested: Availability Results
Running robustly in many large organizations, maintain 99.999%+ availability.
Ultimate business continuity: rolling upgrades, switchovers, and online migrations
No single points of failure: Redundant HA designs for all critical components
Zero-downtime migration based on logical replication blue-green deployment
Emergency plans: standard SOPs for various failure scenarios
Benchmark case: 25,000 vCPU x six years x 99.999% availability
6.3 - Observable Graphics
Clarity and Vision
Heaven’s movement, all-seeing view, perceiving details to master the whole!
Time-series database: Prometheus / AlertManager / X Exporters
Visualization: Grafana, Echarts, plus ten extra panels/data sources
Log collection: Loki / Promtail
Data-Driven: Measure what you Manage
Monitoring built with data analytics and BI principles, turn metrics into insights
From macro overview to micro details, solid data support for administration
3000+ metrics, 666 pre-agg rules, collecting all possible observation points
Unified labels for cross correlation, multi-level dashboard with nav links
Historical metrics of tables, indexes, functions, queries and other DB objects
pg_exporter for postgres/pgbouncer metrics, fully customizable with YAML
Universal Monitoring: RDS or Compat Kernels
Monitoring cloud RDS PG or existing PG-compatible kernels
Monitoring nodes, dbms, apps, LB with correlation analysis
Monitoring cloud vendors’ RDS and PG-compatible kernel like Aurora
Built-in support for: Node, Infra, MinIO, Etcd, Redis, Haproxy, Nginx…
General monitoring solution for servers and other app, easy to integrate
Automatic Alerts: No more manual checks
Production-tested preset alert rule sets with automatic event notification
AlertManager integration: aggregation, silencing, IM connectivity
AlertManager integration: aggregation, silencing, IM connectivity
56 preset alert rules covering all built-in Pigsty modules
Alert info links to dashboards, improve fault diagnosis efficiency
Performance Tuning: Slow-query Optimize
Combine metrics & catalogs to identify and optimize slow queries quickly
Dashboards for query details, with quantifiable comparison results
pg_stat_statements for historical slow query key metrics
auto_explain records slow query execution plans and visualizes with PEV
Use PGCAT to read and visualize PG catalog related data
Log Analysis: Fast root-cause detection
Unified log collection system with Loki and Promtail, sys db log in one place
Search and filter logs with massive parallel grep, fast and efficient
All log streams in one place, associated through labels, flexible search criteria
Use PromQL to extract additional metrics from logs, such as error rates
Use PGLOG application for interactive analysis of specific PG CSV log samples
Custom Dashboards: Low-code Data App
PostgreSQL + Grafana + Echarts + VolkovLabs datasources/panels
Create interactive data app, quickly build prototypes and visualization demos
WHO COVID-19 pandemic data visualization
ISD global landmark weather station data visualization
StackOverflow database file survey
6.4 - Scalable Service
Elastic Performance
Ever-flowing like water, soft yet resilient, converging streams to adapt to endless change!
Great Performance
Hardware Fully Harnessed
R/W Separation
Unlimited Read Scaling
Connection Pooling
High Concurrency
Load Balancing
Traffic Control
Horizontal Scaling
Distributive Extension
Storage Expansion
Transparent Compression
Mass Deployment
Large clusters made easy
Elasticity
Cloud-like Elasticity
Great Performance: Hardware Fully Harnessed
Highlights amazing scalability and top-tier hardware performance.
Maximizing the potential of modern hardware with optimized configurations.
Single-node query rate reaches 2 million rows/second
Single-node write rate can reach 1 million rows/second
Default table size limit: 32TB (2^32 x 8KiB Page)
R/W Separation: Unlimited Read Scaling
Offers unlimited replicas through cascading replication with auto traffic routing.
Scale read workloads horizontally without impacting write performance.
Read-only Service: Route to read-only replicas with primary as backup
Offline Service: Route to special analytics instance with replicas as backup
Production Case: One primary with 34+ replicas through cascading bridges
Connection Pooling: High Concurrency
Built-in PGBouncer connection pool, ready out of the box and syncing with postgres.
Handle thousands of concurrent connections efficiently with resource optimization.
Xact pooling converts 20000+ client connections to several active server connections
Enabled by default, automatically syncing db/user with postgres
Deploy multiple pgbouncer instances to circumvent bottlenecks
Load Balancing: Traffic Control
Monitor and schedule request traffic in real-time with HAProxy console.
Intelligent traffic distribution with health checks and failover capabilities.
Stateless HAProxy can be scaled at will or deployed on dedicated servers
Weights can be adjusted via CLI, draining or warming up instances gracefully
Password-protected HAProxy GUI exposed uniformly through Nginx
Horizontal Scaling: Distributive Extension
Citus extension with multi-write and multi-tenant capabilities.
Scale beyond single-node limitations with distributed PostgreSQL.
Accelerate real-time OLAP analytics using multi-node parallel processing
Shard by row key or schema, supporting multi-tenant scenarios
Achieve 10:1 or even higher compression ratios with columnar and other exts
R/W data in S3 with FDW, hot/cold separation and unlimited capacity expansion
Use timescaledb, pg_mooncake, pg_duckdb for columnar compression
Use duckdb_fdw, pg_parquet, pg_analytics to read/write object storage tables
Expand or contract storage with S/H RAID, ZFS, and PG tablespaces
Mass Deployment: Large clusters made easy
Designed for extreme scale - flexible for 25K vCPU clusters or 1c1m node
No limit on nodes per deployment - soft constrained only by monitoring capacity
Batch Ops at scale through Ansible, saying goodbye to console ClickOps
Largest production deployment record: 25,000 vCPU, 3,000+ instances
Elasticity: Cloud-like Elasticity
Supports cloud EC2 deployment, fully leveraging the elastic advantages of cloud
Flexible multi-cloud strategies - enjoy RDS elasticity with EC2/EBS prices
Pigsty only needs cloud servers, works the same across any cloud provider
Seamless switching between public, private, hybrid, and multi-cloud
Scale compute and storage as needed
6.5 - Maintainable Toolbox
Simple and Actionable
Blazing like wildfire, illuminating all around, burning bright without end!
Infra as Code
Define everything in Code
Simple & Easy
Up and Running in Minutes
Bare Linux
No Containers or Kubernetes
Offline Install
Stable, Hassle-free Delivery
Admin SOP
Best Practices Included
No Downtime
Online Migration & Resizing
Rich Parameters
Plenty of Tunable Knobs
Provisioning
One-command IaaS Provisioning
Infra as Code: Define everything in Code
Declarative API for database deployment and operations
Designed for large-scale cluster management
Define databases through config files
Designed for batch operations
Optional PostgreSQL CMDB solution for programmatic integration
Simple & Easy: Up and Running in Minutes
One-command installation
Pre-configured templates with automatic hardware tuning
Config templates for various node counts and scenarios
Runs on single-core VMs to high-end physical machines
Logical Replication, Decoding, CDC in protobuf/JSON/Mongo format, Copy & Load & Compare Postgres Databases
Repository
Pigsty has a repository that provides 200+ extra PostgreSQL extensions on 10 mainstream Linux Distros.
It is designed to work together with the official PostgreSQL Global Development Group (PGDG) repo.
You can enable the pigsty infra & pgsql repo with the pig CLI tool, or add them manually to your system:
curl https://repo.pigsty.io/pig | bash # download and install the pig CLI toolpig repo add all -u # add linux, pgdg, pigsty repo and update cache
# Add Pigsty's GPG public key to your system keychain to verify package signaturescurl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg
# Get Debian distribution codename (distro_codename=jammy, focal, bullseye, bookworm), and write the corresponding upstream repository address to the APT List filedistro_codename=$(lsb_release -cs)sudo tee /etc/apt/sources.list.d/pigsty-io.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql/${distro_codename} ${distro_codename} main
EOF# Refresh APT repository cachesudo apt update
# Add Pigsty's GPG public key to your system keychain to verify package signaturescurl -fsSL https://repo.pigsty.io/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
# Add Pigsty Repo definition files to /etc/yum.repos.d/ directory, including two repositoriessudo tee /etc/yum.repos.d/pigsty-io.repo > /dev/null <<-'EOF'
[pigsty-infra]
name=Pigsty Infra for $basearch
baseurl=https://repo.pigsty.io/yum/infra/$basearch
skip_if_unavailable = 1
enabled = 1
priority = 1
gpgcheck = 1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty
module_hotfixes=1
[pigsty-pgsql]
name=Pigsty PGSQL For el$releasever.$basearch
baseurl=https://repo.pigsty.io/yum/pgsql/el$releasever.$basearch
skip_if_unavailable = 1
enabled = 1
priority = 1
gpgcheck = 1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty
module_hotfixes=1
EOF# Refresh YUM/DNF repository cachesudo yum makecache;
All the RPM / DEB packages are signed with GPG Key fingerprint (B9BD8B20) in Pigsty repository.
Package Manager
“Postgres Install Genius, the missing extension package manager for PostgreSQL ecosystem”
Simulate other DBMS, and replace vanilla PostgreSQL with exotic forks
Pigsty supports various PostgreSQL kernels and compatible forks,
enabling you to simulate different database systems while leveraging PostgreSQL’s ecosystem.
Each kernel provides unique capabilities and compatibility layers.
Database Kernels
PostgreSQL
Vanilla Postgres with 437 Extensions
Citus
Native Distributive Extension
Babelfish
SQL Server wire-compatible
IvorySQL
Oracle grammar & PL/SQL compatible
OpenHalo
MySQL wire-compatibility
Percona
Transparent Data Encryption
OrioleDB
OLTP-optimized cloud-native storage engine
PolarDB PG
Aurora-like RAC with china domestic compliance
Supabase
Backend as a Service, self-hosting Firebase
FerretDB
Mongo Wire-Compatibility over PostgreSQL
Choose the Right Kernel
Note
Flexible Kernel: Choose the right kernel for your specific use case - whether you need MSSQL compatibility, Oracle features, or horizontal scaling capabilities.
Citus transforms PostgreSQL into a distributed database system, enabling horizontal scaling across multiple nodes.
Deploy native HA Citus clusters with Pigsty for better throughput and performance.
Key Features
Distributed Tables: Automatically shard tables across worker nodes
Distributed Queries: Execute queries across the entire cluster
High Availability: Built-in replication and failover capabilities
Real-time Analytics: Handle both transactional and analytical workloads
Postgres Compatibility: Maintain full PostgreSQL feature compatibility
Self-host Supabase with existing managed HA PostgreSQL clusters, launching the stateless components with docker-compose for a complete Firebase alternative.
Key Features
Real-time APIs: Auto-generated REST and GraphQL APIs
Real-time Subscriptions: WebSocket-based real-time data sync
Authentication: Built-in user authentication and authorization
Storage: File storage with CDN capabilities
Edge Functions: Serverless functions for custom logic
Use Cases
Rapid application development with backend-as-a-service
Real-time applications requiring instant data sync
JAMstack applications needing serverless backend
Mobile and web apps require authentication and storage
Note
Full Stack: Provides a complete backend solution with PostgreSQL as the foundation.
Install and monitor Greenplum / Cloudberry / YMatrix MPP clusters with Pigsty for large-scale analytical processing and data warehousing.
Key Features
Massively Parallel Processing: Distribute queries across multiple nodes
Columnar Storage: Optimized storage for analytical workloads
Advanced Analytics: Built-in machine learning and statistical functions
Petabyte Scale: Handle massive datasets with linear scalability
Standard SQL: Full SQL compliance with PostgreSQL compatibility
Use Cases
Data warehousing and business intelligence
Large-scale analytics and reporting
Machine learning on big datasets
ETL processing for enterprise data platforms
Note
Enterprise Analytics: Designed for enterprise-scale analytical workloads requiring massive parallel processing capabilities.
7.3 - Observability Infra
Modern observability stack with 3000+ metrics, 30+ dashboards, and enterprise-grade monitoring
Pigsty delivers unparalleled observability with a modern monitoring stack built on industry best practices.
Automatically monitor every component with 3000+ metrics, 30+ dashboards.
Note
Complete Insight: Monitor everything from high-level cluster health to individual table statistics. Get complete insight into the past, present, and future of your infrastructure.
Architecture Overview
Pigsty’s observability infrastructure leverages battle-tested open-source components in a cohesive, production-ready stack:
Grafana Visualization Engine
Dashboards with advanced interactive visualizations
Purpose: Rich, interactive visualizations for complex data analysis
Data Sources
# Extended data source supportgrafana_datasources:- infinity-datasource # REST API and file data sources- redis-datasource # Redis data source- clickhouse-datasource # ClickHouse integration- postgres-datasource # Enhanced PostgreSQL support
Purpose: Connect to diverse data sources beyond traditional metrics
Advantages: Flexible, lightweight, and highly configurable
Version Compatibility
# PostgreSQL version support matrixsupported_versions:- postgresql_9_6:legacy_metrics_set- postgresql_10:enhanced_metrics_set- postgresql_11:advanced_metrics_set- postgresql_12:modern_metrics_set- postgresql_13:extended_metrics_set- postgresql_14:latest_metrics_set- postgresql_15:cutting_edge_metrics_set- postgresql_16:next_gen_metrics_set
Benefit: Single exporter for heterogeneous PostgreSQL environments
Custom Queries
# Custom metric definition examplecustom_queries:pg_custom_business_metrics:query:| SELECT
schemaname,
tablename,
n_tup_ins as inserts_total,
n_tup_upd as updates_total,
n_tup_del as deletes_total
FROM pg_stat_user_tablesmetrics:- inserts_total:usage:COUNTERdescription:"Total number of inserts"- updates_total:usage:COUNTERdescription:"Total number of updates"
Pigsty v3.7 installs node_exporter on managed nodes and registers Node,
HAProxy, Keepalived, Nginx, Etcd, MinIO, Redis, PostgreSQL, PgBouncer, and
pgBackRest targets with Prometheus when the corresponding modules are
enabled. Component ports and switches are defined by the tagged role
defaults and documented on each module’s parameter page.
External Database Monitoring
Existing PostgreSQL instances and cloud RDS can be registered with
pgsql-monitor.yml. Supply a monitor connection URL with only the required
pg_monitor privileges; Pigsty registers pg_exporter and the Grafana
datasource without provisioning or changing that database.
Grafana can query both Prometheus metrics and PostgreSQL data. The bundled
dashboards use variables and URL links for drill-down navigation; ECharts
and other packaged plugins can visualize application or business data.
Low-Code Application Development
Grafana panels may be combined with PostgreSQL queries to build internal
operational views. This is a visualization capability, not a separate
Pigsty deployment module or application API.
Reusable Infrastructure
infra.yml can deploy INFRA independently from PGSQL. Nginx, DNSMasq,
Prometheus, AlertManager, Grafana, and Loki are controlled by their
documented *_enabled parameters, allowing Pigsty to reuse or coexist with
existing infrastructure.
Best Practices
Control metric cardinality and set retention according to available disk.
Back up Grafana configuration and keep alert notification channels tested.
Protect monitoring endpoints with access controls and HTTPS when exposed
outside the trusted network.
Monitor the monitoring stack itself and review storage growth regularly.
Limitations & Considerations
Prometheus and Loki in v3.7 are single-node services by default. Very large
metric cardinality, long retention, or complex dashboards require capacity
planning. External long-term storage and HA monitoring architectures are
manual integrations rather than built-in v3.7 modules.
7.4 - High Availability
Enterprise-grade high availability with automatic failover and self-healing capabilities
Pigsty uses Patroni to achieve high availability for PostgreSQL, ensuring automatic failover.
Pigsty’s PostgreSQL cluster has battery-included high-availability powered by Patroni, Etcd, and HAProxy.
When you have two or more instances in the PostgreSQL cluster, you have the ability to self-heal from hardware failures without any further configuration — as long as any instance within the cluster survives, the cluster can serve its services. Clients simply need to connect to any node in the cluster to obtain full services without worrying about replication topology changes.
By default, the recovery time objective (RTO) for primary failure is approximately 30s ~ 60s, and the data recovery point objective (RPO) is < 1MB; for standby failure, RPO = 0, RTO ≈ 0 (instantaneous). In consistency-first mode, zero data loss during failover is guaranteed: RPO = 0. These metrics can be configured as needed based on your actual hardware conditions and reliability requirements.
Pigsty incorporates an HAProxy load balancer for automatic traffic switching, offering multiple access methods for clients such as DNS/VIP/LVS. Failovers and switchover are almost imperceptible to the business side except for sporadic interruptions, meaning applications do not need connection string modifications or restarts.
Key Metrics
RTO ~ 30s
Primary Failure
RPO < 1MB
Async Mode RPO
RTO ~ 0s
Replica Failure
RPO = 0
Sync Mode
What High Availability Solves
High availability addresses critical operational challenges:
Data Safety
Elevates availability: RPO ≈ 0, RTO < 30s for enhanced data protection
Rolling Maintenance
Seamless maintenance: Minimize maintenance windows for operational convenience
Hardware Failures
Self-healing: Automatic recovery from hardware failures without human intervention
Load Distribution
Read scaling: Distribute read-only queries across standby instances
Specific Benefits
Enhanced data safety: Improves the availability aspect of data safety CIA to a new height
Rolling maintenance capabilities: Enables seamless maintenance with minimal downtime
Hardware failure recovery: Self-healing from hardware failures without human intervention
Load sharing: Read-only requests can be distributed across standby instances
Costs of High Availability
Implementing HA introduces certain trade-offs and requirements:
Note
Infrastructure Requirements: HA requires at least 3 nodes and additional infrastructure dependencies.
Resource Requirements
Minimum cluster size: At least 3 nodes for proper consensus
Additional infrastructure: Requires consensus store (Etcd) and load balancer
Resource overhead: Additional CPU, memory, and network resources
Operational complexity: Increased monitoring and management requirements
Limitations
Note
High availability cannot prevent:
Human errors and operational mistakes
Software defects causing data corruption
Logical data deletion or corruption
For these scenarios, additional recovery strategies are needed:
Network conditions significantly affect HA behavior:
High-quality networks: Can use lower RTO values safely
Unstable networks: Require higher RTO to prevent false positives
WAN deployments: Need careful tuning of timeout parameters
Local networks: Can optimize for faster failover
Monitoring and Observability
Pigsty provides comprehensive monitoring for HA cluster health:
Key Metrics
Cluster State
Monitor cluster topology, leader status, and member health
Replication Lag
Track replication lag and sync status across all replicas
Failover Events
Log and analyze failover events and their impact
Performance
Monitor query performance and connection health
Dashboard Integration
Pigsty includes pre-built Grafana dashboards for HA monitoring:
Cluster Overview: Real-time cluster topology and health
Replication Monitoring: Lag metrics and sync status
Failover Analysis: Historical failover events and timing
Performance Metrics: Query performance during normal and failover scenarios
Best Practices
Deployment Recommendations
Note
Anti-Affinity: Deploy cluster nodes across different physical hosts, racks, or availability zones.
Hardware diversity: Use different hardware configurations to avoid common failure modes
Network redundancy: Ensure multiple network paths between cluster nodes
Storage considerations: Use local storage for best performance, shared storage for specific use cases
Monitoring setup: Implement comprehensive monitoring before going to production
Operational Guidelines
Regular testing: Perform controlled failover tests in non-production environments
Capacity planning: Size cluster nodes appropriately for failover scenarios
Backup strategy: Maintain regular backups independent of HA setup
Documentation: Keep runbooks updated for emergency procedures
Common Pitfalls
Note
Avoid These Common Mistakes:
Insufficient network bandwidth between nodes
Inadequate monitoring of replication lag
Not testing failover procedures regularly
Incorrect firewall configurations
Summary
Pigsty’s high availability solution provides:
Automatic failover with sub-minute RTO
Configurable consistency with RPO control
Self-healing capabilities for hardware failures
Load balancing for read scaling
Minimal operational overhead with automated management
The combination of Patroni, Etcd, and HAProxy creates a robust, production-ready HA solution that handles the majority of failure scenarios automatically while providing the flexibility to tune behavior based on specific requirements.
High availability is not just about technology—it’s about building resilient systems that your business can depend on.
7.5 - Disaster Recovery
Battle-tested backup strategies and auto-configured Point-in-Time Recovery
Point-in-Time Recovery (PITR) allows rolling back a PostgreSQL cluster to any specific moment in the past, preventing data loss from software defects or human errors. Pigsty uses pgBackRest for PITR, with configurable backup strategies using local filesystems or object storage like MinIO.
Note
Time Travel for Databases: Roll back your cluster to any point in time, protecting against software defects, human errors, and data corruption scenarios that high availability cannot address.
Overview
Pigsty provides enterprise-grade Point-in-Time Recovery with zero-configuration setup, automated backups, and flexible restore options. Built on pgBackRest with MinIO/S3 support, it protects against data corruption, human errors, and logical disasters.
Reduced RPO
Minimize Recovery Point Objective with continuous WAL archiving
Data Integrity
Enhanced data integrity protection against corruption
Disaster Recovery
Improved disaster recovery capabilities with flexible restore options
How PITR Works
PITR requires two key components working together to enable point-in-time recovery:
Base Backups
Base Backups
Uses pgBackRest to create database cluster snapshots with multiple backup types:
- **Full backups**: Complete database cluster snapshots
- **Incremental backups**: Only changes since the last backup
- **Differential backups**: Changes since the last full backup
- **Scheduled backups**: Periodic backups configured via Crontab
WAL Archiving
WAL Archiving
Continuously archives Write-Ahead Log (WAL) segment files:
- **Continuous archiving**: Real-time WAL file preservation
- **Automatic management**: WAL files and cleanup handled automatically
- **Optional feature**: Can be disabled if PITR is not needed
Implementation
Pigsty provides two default backup strategies with flexible configuration options:
Local Filesystem
Local Filesystem Strategy
- **Frequency**: Daily full backups
- **Storage**: Local filesystem storage
- **Use case**: Single-node or local development environments
MinIO/S3
MinIO/S3 Strategy
- **Frequency**: Weekly full backup with daily incremental backups
- **Storage**: Object storage (MinIO, S3)
- **Use case**: Production environments with distributed storage
Configuration Options
Backup configuration is highly flexible with options to specify:
Repository type: Local, S3, or other supported backends
Retention policies: How long to keep backups
Encryption: Secure backup storage
Storage locations: Multiple backup destinations
Recovery Options
Note
Recovery operations should be performed carefully as they will replace the current database state.
Recovery commands allow restoring to various points in time:
Step 1
### Latest WAL Archive
Restore to the most recent point available in WAL archives:
```bash
pg-pitr
```
Step 2
### Specific Timestamp
Restore to an exact moment in time:
```bash
pg-pitr --time="2022-12-30 14:44:44+08"
```
Step 3
### Named Restore Point
Restore to a previously created named point:
```bash
pg-pitr --name="my-restore-point"
```
Step 4
### Specific LSN or Transaction ID
Restore to a particular Log Sequence Number or transaction:
```bash
pg-pitr --lsn="0/1234567"
pg-pitr --xid="12345"
```
Note
While PITR is powerful for data recovery, it should ideally be combined with High Availability solutions for comprehensive data protection against both logical and physical failures.
7.6 - Infra as Code
Declarative infrastructure and database management with YAML-driven configuration
Pigsty provides a declarative interface: Describe everything in a config file, and Pigsty operates it to the desired state with idempotent playbooks. It works like Kubernetes CRDs & Operators but for databases and infrastructures on any nodes: bare metal or virtual machines.
Note
Infra as Code, Database as Code: Declarative API & Idempotent Playbooks, GitOPS works like a charm.
Declare Module
You can declare modules on a single node:
# infra cluster for proxy, monitor, alert, etc...infra:{hosts:{10.10.10.10:{infra_seq:1}}}# minio cluster, s3 compatible object storageminio:{hosts:{10.10.10.10:{minio_seq:1 } }, vars:{minio_cluster:minio } }# etcd cluster for ha postgres DCSetcd:{hosts:{10.10.10.10:{etcd_seq:1 } }, vars:{etcd_cluster:etcd } }# postgres example cluster: pg-metapg-meta:{hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }, vars:{pg_cluster:pg-meta } }
And apply with playbooks:
./infra.yml -l infra # init infra module on node 10.10.10.10./etcd.yml -l etcd # init etcd module on node 10.10.10.10./minio.yml -l minio # init minio module on node 10.10.10.10./pgsql.yml -l pg-meta # init pgsql module on node 10.10.10.10
Declare Cluster
To create a three-node HA postgres cluster with streaming replication:
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_databases:- name:metabaseline:cmdb.sqlcomment:pigsty meta databaseschemas:[pigsty]extensions:- {name:adminpack, schema:pg_catalog }- {name:postgis, schema:public }- {name:timescaledb, schema:public }pg_users:- {name:dbuser_meta, password:DBUser.Meta, pgbouncer:true, roles:[dbrole_admin], comment:pigsty admin user }- {name:dbuser_view, password:DBUser.Viewer, pgbouncer:true, roles:[dbrole_readonly], comment:pigsty read-only user }pg_services:- {name:primary, port:5433, dest:default }- {name:replica, port:5434, dest:default, selector:"[]"}- {name:default, port:5436, dest:postgres }- {name:offline, port:5438, dest:postgres, selector:"[]"}pg_hba_rules:- {user:dbuser_view, db:all, addr:infra, auth:pwd, title:'allow view user from infra nodes'}pgb_hba_rules:- {user:dbuser_view, db:all, addr:infra, auth:pwd, title:'allow view user from infra nodes'}
Declare Access Control
Define advanced access control rules:
pg_hba_rules:- {user:'${dbsu}', db:all, addr:local, auth:ident, title:'dbsu access via local os user ident'}- {user:'${dbsu}', db:replication, addr:local, auth:ident, title:'dbsu replication from local os ident'}- {user:'${repl}', db:replication, addr:'${ip}/32', auth:pwd, title:'replicator replication from ${ip}'}- {user:'${repl}', db:postgres, addr:'${ip}/32', auth:pwd, title:'replicator postgres db from ${ip}'}- {user:'${monitor}', db:all, addr:'${ip}/32', auth:pwd, title:'monitor from ${ip}'}- {user:'${monitor}', db:all, addr:infra, auth:pwd, title:'monitor from infra nodes'}- {user:'${admin}', db:all, addr:infra, auth:ssl, title:'admin @ infra nodes with pwd & ssl'}- {user:'+dbrole_readonly', db:all, addr:'${vip}/32', auth:ssl, title:'allow readonly role from ${vip} with ssl'}- {user:'+dbrole_offline', db:all, addr:'${vip}/32', auth:ssl, title:'allow offline role from ${vip} with ssl'}- {user:dbuser_meta, db:meta, addr:'10.0.0.0/8', auth:ssl, title:'allow meta user from 10.0.0.0/8 with ssl'}
Pigsty enables you to describe your entire infrastructure declaratively and manage it through code, providing consistency, repeatability, and scalability for your database and infrastructure operations.
7.7 - No Containers
Pigsty runs on raw Linux without containers and kubernetes
Pigsty runs on bare Linux, We support mainstream Linux distributions like EL / Debian / Ubuntu, and compatible Linux distributions like Rocky Linux, AlmaLinux, etc…
We do this on purpose. It is hard to compile and package all the postgres related packages and hundreds of extensions into RPM/DEB for each Major version x PG version x OS Arch…
But I believe this is the right thing to do. So we don’t take shortcuts like Docker, Podman, or Kubernetes.
7.8 - App Templates
Setup enterprise software with templates, supabase, odoo, dify, gitlab,…
Supabase
Self-Hosting Supabase
Odoo
Run Odoo the OSS ERP
Dify
Run Dify the AI workflow
pgAdmin
Run official Admin GUI Tool
7.9 - Local First
Install without Internet, All dependencies included
7.10 - Supabase
Turn Postgres into a full-featured backend as a service
Nowadays, people use OS distributions like Ubuntu, Debian, and RHEL, rather than the raw Linux kernel directly.
You’ll need a lot of components to build a practical operating system, such as systemd, cron, NTP, DNS, logging, …, to make the raw Linux kernel usable.
The linux kernel is several MB in size, but a full OS DVD can easily take up 10+GB, including all the necessary parts and software packages.
That’s what a PostgreSQL Distribution is all about —— Gives you everything to build a production-grade Database Service.
Why do we need Distribution?
We have two things to forge a powerful PostgreSQL distribution: Extensions and Services.
Extensions
There are 1000+ extensions in the PostgreSQL ecosystem.
But only 100 of them are accessible via the “Official” PGDG Repositories.
So we packed the most popular and useful extensions into pre-made RPM/DEB packages for 10 linux distributions and 5 PG Major version.
Now there are unparalleled 422 extensions available out-of-the-box, and we will continue to add more extensions in the future.
What’s more, we even have support for 8 flavors of PostgreSQL kernels (ext, fork, wrapper, etc.), including:
You can get started with raw PostgreSQL kernel easily like systemctl start postgresql, but it’s far away from production grade service.
That’s the main reason why people pay 160 $ / vCPU·Month for a managed PostgreSQL service like AWS RDS.
But what if you can just build an enterprise-grade PostgreSQL service on your own, with a few commands, and no license fees?
Pigsty enables you to do that. It gives you HA PostgreSQL clusters with PITR, Monitoring & Alerting, Connection Pooling, along with
8.2 - RDS Alternative
Self-hosting PostgreSQL like a Pro! Operate production grade Service without expertise
What is RDS?
You can get started with raw PostgreSQL kernel easily like systemctl start postgresql, but it’s far away from production grade service.
That’s the main reason why people pay 160 $ / vCPU·Month for a managed PostgreSQL service like AWS RDS. and even pay more for traditional “enterprise” database services.
The expertise of Building and Managing Production-Grade PostgreSQL Service is rare and expensive.
What if…
But what if you can just build an enterprise-grade PostgreSQL service on your own, with a few commands, and no license fees?
Pigsty enables you to do that. It gives you HA PostgreSQL clusters with PITR, Monitoring & Alerting, Connection Pooling, along with
It’s all starts from a few commands, and you can build a production-grade PostgreSQL service on your own,
without the need for expensive licenses or expertise.
8.3 - Architecture
Pigsty’s modular, declarative infrastructure design for PostgreSQL
Modular Architecture and Declarative Interface!
Pigsty deployment is described by config inventory and materialized with ansible playbooks.
Pigsty works on Linux common nodes, i.e., bare metals or virtual machines.
Pigsty uses a modular design that can be freely composed for different scenarios.
The config controls where & how to install modules with parameters
The playbooks will adjust nodes into the desired status in an idempotent manner.
ETCD: Distributed key-value store will be used as DCS for high-available Postgres clusters.
REDIS: Redis servers in standalone master-replica, sentinel, cluster mode with Redis exporter.
MINIO: S3 compatible simple object storage server, can be used as an optional backup center for Postgres.
You can compose them freely in a declarative manner. If you want host monitoring, INFRA & NODE will suffice. Additional ETCD and PGSQL are used for HA PG Clusters. Deploying them on multiple nodes will form an HA cluster. You can reuse pigsty infra and develop your modules, considering optional REDIS and MINIO as examples.
Singleton Meta
Pigsty will install on a single node (BareMetal / VirtualMachine) by default. The install.yml playbook will install INFRA, ETCD, PGSQL, and optional MINIO modules on the current node, which will give you a full-featured observability infrastructure (Prometheus, Grafana, Loki, AlertManager, PushGateway, BlackboxExporter, etc… ) and a battery-included PostgreSQL Singleton Instance (Named meta).
This node now has a self-monitoring system, visualization toolsets, and a Postgres database with autoconfigured PITR. You can use this node for devbox, testing, running demos, and doing data visualization & analysis. Or, furthermore, adding more nodes to it!
Monitoring
The installed Singleton Meta can be used as an admin node and monitoring center, to take more nodes & Database servers under it’s surveillance & control.
If you want to install the Prometheus / Grafana observability stack, Pigsty just deliver the best practice for you! It has fine-grained dashboards for Nodes & PostgreSQL, no matter these nodes or PostgreSQL servers are managed by Pigsty or not, you can have a production-grade monitoring & alerting immediately with simple configuration.
HA PG Cluster
With Pigsty, you can have your own local production-grade HA PostgreSQL RDS as much as you want.
And to create such a HA PostgreSQL cluster, All you have to do is describe it & run the playbook:
Which will give you the following cluster with monitoring, replica, backup all set.
Hardware failures are covered by self-healing HA architecture powered by patroni, etcd, and haproxy, which will perform auto failover in case of leader failure under 30 seconds. With the self-healing traffic control powered by haproxy, the client may not even notice there’s a failure at all, in case of a switchover or replica failure.
Software Failures, human errors, and DC Failure are covered by pgbackrest, and optional MinIO clusters. Which gives you the ability to perform point-in-time recovery to anytime (as long as your storage is capable)
Database as Code
Pigsty follows IaC & GitOPS philosophy: Pigsty deployment is described by declarative Config Inventory and materialized with idempotent playbooks.
The user describes the desired status with Parameters in a declarative manner, and the playbooks tune target nodes into that status in an idempotent manner. It’s like Kubernetes CRD & Operator but works on Bare Metals & Virtual Machines.
Take the default config snippet as an example, which describes a node 10.10.10.10 with modules INFRA, NODE, ETCD, and PGSQL installed.
# infra cluster for proxy, monitor, alert, etc...infra:{hosts:{10.10.10.10:{infra_seq:1}}}# minio cluster, s3 compatible object storageminio:{hosts:{10.10.10.10:{minio_seq:1 } }, vars:{minio_cluster:minio } }# etcd cluster for ha postgres DCSetcd:{hosts:{10.10.10.10:{etcd_seq:1 } }, vars:{etcd_cluster:etcd } }# postgres example cluster: pg-metapg-meta:{hosts:{10.10.10.10:{pg_seq:1, pg_role:primary }, vars:{pg_cluster:pg-meta } }
To materialize it, use the following playbooks:
./infra.yml -l infra # init infra module on group 'infra'./etcd.yml -l etcd # init etcd module on group 'etcd'./minio.yml -l minio # init minio module on group 'minio'./pgsql.yml -l pg-meta # init pgsql module on group 'pgsql'
It would be straightforward to perform regular administration tasks. For example, if you wish to add a new replica/database/user to an existing HA PostgreSQL cluster, all you need to do is add a host in config & run that playbook on it, such as:
How Pigsty compares to cloud RDS, Kubernetes operators, and other PostgreSQL solutions
Notice: this post is outdated and re-generated with claude
Pigsty positions itself as a local-first, open-source PostgreSQL platform that challenges
traditional cloud database services and complex orchestration platforms.
This comparison demonstrates Pigsty’s advantages across key dimensions.
Cloud RDS Comparison
AWS RDS PostgreSQL vs Pigsty
Feature
AWS RDS PostgreSQL
Pigsty
Deployment
Managed cloud service
Self-hosted on bare metal/VM/cloud
Cost
$200-1,300/core/month
$20-40/core/month hardware cost
Licensing
Proprietary + usage fees
AGPLv3 open source
Extensions
Limited, AWS-approved only
400+ extensions freely available
Monitoring Metrics
99 basic metrics
3,000+ comprehensive metrics
Dashboards
CloudWatch basic views
50+ specialized dashboards
Superuser Access
Restricted
Full superuser privileges
Data Sovereignty
AWS controlled
Complete local control
Offline Operation
Impossible
Full offline capability
Migration Flexibility
Vendor lock-in
Multi-cloud portability
Cost Analysis
Traditional Cloud RDS Pricing:
AWS RDS: $1,920-$2,640/vCPU/year
Alibaba Cloud: $200-1,300/core/month
Azure Database: Similar premium pricing
Pigsty Total Cost of Ownership:
Hardware: $27/vCPU/year
Software: Free (open source)
Savings: 50-95% compared to cloud RDS
Observability Advantage
Pigsty Monitoring Capabilities:
3,000+ metrics vs cloud providers’ 99-200 metrics
638 PostgreSQL-specific metrics for deep database insights
50+ pre-built dashboards covering all infrastructure layers
Real-time query analysis and performance tuning
Custom dashboard creation with low-code tools
Cloud Provider Limitations:
Basic CloudWatch/Azure Monitor metrics
Limited customization options
Additional costs for enhanced monitoring
No access to underlying system metrics
Kubernetes Operators Comparison
Traditional K8s Operators vs Pigsty
Aspect
Kubernetes Operators
Pigsty
Complexity
High learning curve
Simple Ansible-based
Dependencies
Kubernetes cluster required
Bare Linux sufficient
Resource Overhead
Container orchestration overhead
Native performance
Monitoring
Separate monitoring stack needed
Integrated observability
Storage
Complex PV/PVC management
Direct storage access
Networking
K8s networking complexity
Standard Linux networking
Debugging
Multi-layer troubleshooting
Direct system access
Operational Burden
Kubernetes + DB operations
Database-focused operations
Why Choose Pigsty Over K8s Operators
Simplicity Benefits:
No container orchestration complexity
Direct hardware access for optimal performance
Familiar Linux tools for debugging and maintenance
Reduced attack surface without container layers
Operational Advantages:
Lower resource overhead compared to containerized solutions
Easier troubleshooting with direct system access
Simpler backup/restore operations
Native OS integration for security and monitoring
PostgreSQL Distributions Comparison
Commercial Distributions
Feature
EnterpriseDB
Postgres Pro
VMware Postgres
Pigsty
Licensing
Commercial
Commercial
Commercial
AGPLv3
Cost
High license fees
High license fees
High license fees
Free
Extensions
Limited selection
Curated set
VMware-specific
400+ available
Monitoring
Additional purchase
Basic included
vCenter integration
3,000+ metrics included
High Availability
Enterprise feature
Available
Available
Built-in
Support
Paid support only
Paid support
VMware support
Community + commercial
Open Source Alternatives
Solution
Focus Area
Pigsty Advantage
Patroni
HA clustering only
Complete platform with monitoring
PostgreSQL Helm Charts
K8s deployment
No K8s dependency, simpler ops
Postgres Operator
K8s orchestration
Native performance, easier management
TimescaleDB Cloud
Time-series focus
General-purpose with time-series support
Supabase
Backend-as-a-Service
Full infrastructure control
Multi-Cloud Strategy
Vendor Lock-in Avoidance
Cloud Provider Lock-in Risks:
Proprietary APIs and tooling
Data transfer costs for migration
Feature dependency on specific platforms
Pricing changes and service discontinuation
Pigsty Multi-Cloud Benefits:
Portable across any Linux environment
Consistent operations regardless of underlying infrastructure
Freedom to negotiate with cloud providers
Hybrid deployment capabilities
Migration Flexibility
graph TB
A[Existing PostgreSQL] --> B[Pigsty Migration Tool]
B --> C[Cloud Provider A]
B --> D[Cloud Provider B]
B --> E[On-Premises]
B --> F[Hybrid Deployment]
C --> G[Cross-Cloud Replication]
D --> G
E --> G
F --> G
Migration Capabilities:
Logical replication for live migration
Point-in-time recovery across environments
Configuration portability via Infrastructure as Code
Zero-downtime migration procedures
Extension Ecosystem
Extension Availability Comparison
Category
Cloud RDS
Pigsty
Analytics
Limited (no pg_duckdb)
Full OLAP stack
Vector/AI
Basic pgvector
pgvector, pgml, pg_embedding
Geospatial
PostGIS only
PostGIS + advanced GIS extensions
Time Series
Basic TimescaleDB
TimescaleDB + specialized tools
Graph
Not available
Apache AGE + graph extensions
Search
Basic text search
Advanced search + vector hybrid
Monitoring
None
pg_stat_monitor + custom metrics
Enterprise Extensions
Pigsty Includes:
pg_duckdb: Extreme analytics performance
PostgresML: In-database machine learning
Citus: Distributed PostgreSQL
PostGIS: Advanced geospatial capabilities
pgvector: Vector similarity search
TimescaleDB: Time-series optimization
Cloud Limitations:
Restricted extension catalog
Version dependencies
Installation restrictions
Limited configuration options
Performance Characteristics
Hardware Optimization
Pigsty Performance Advantages:
Direct hardware access without virtualization overhead
Custom kernel tuning for database workloads
NUMA awareness and CPU affinity optimization
Storage optimization for specific workload patterns
Benchmark Results:
2M rows/second query throughput on optimized hardware
Existing infrastructure with available hardware resources
When to Consider Alternatives
Cloud RDS Advantages:
Minimal operational overhead for small teams
Quick proof-of-concept development
Geographic distribution without infrastructure management
Compliance certifications already established
Migration Path:
Start with cloud RDS for rapid development
Migrate to Pigsty for production cost optimization
Maintain hybrid deployments for specific use cases
Pigsty delivers enterprise-grade PostgreSQL capabilities with significant cost savings, superior observability, and complete operational control, making it a compelling alternative to cloud database services and complex orchestration platforms.
8.5 - Modules
Available modules in Pigsty
Core Modules
Pigsty consists of multiple modules. The PINE stack: PGSQL / INFRA / NODE / ETCD are ESSENTIAL for self-hosting Postgres RDS service.
PGSQL
HA PG Cluster with PITR, IaC, ACL, Monitor, and 437 extensions
INFRA
Nginx, Repo, DNS, NTP, Prometheus and Grafana stack for Observability
NODE
Enroll nodes into the desired state and monitor it, and VIP, HAProxy
ETCD
Reliable distributive consensus storage (DCS), empowering PGSQL HA
Extra Modules
Pigsty also have some OPTIONAL “Bonus” modules, which works well with PostgreSQL, and brings extra value to your data infrastructure.
Added new pgBackRest backup monitoring metrics and dashboards
Enhanced Nginx server configuration options, with support for automated Certbot issuance
Now prioritizing PostgreSQL’s built-in C/C.UTF-8 locale settings
IvorySQL 4.4 is now fully supported across all platforms (RPM/DEB on x86/ARM)
Added new software packages: Juicefs, Restic, TimescaleDB EventStreamer
The Apache AGE graph database extension now fully supports PostgreSQL 13–17 on EL
Improved the app.yml playbook: launch standard Docker app without extra config
Bump Supabase, Dify, and Odoo app templates, bump to their latest versions
Add electric app template, local-first PostgreSQL Sync Engine
Infra Packages
+restic 0.17.3
+juicefs 1.2.3
+timescaledb-event-streamer 0.12.0
Prometheus 3.2.1
AlertManager 0.28.1
blackbox_exporter 0.26.0
node_exporter 1.9.0
mysqld_exporter 0.17.2
kafka_exporter 1.9.0
redis_exporter 1.69.0
pgbackrest_exporter 0.19.0-2
DuckDB 1.2.1
etcd 3.5.20
FerretDB 2.0.0
tigerbeetle 0.16.31
vector 0.45.0
VictoriaMetrics 1.113.0
VictoriaLogs 1.17.0
rclone 1.69.1
pev2 1.14.0
grafana-victorialogs-ds 0.16.0
grafana-victoriametrics-ds 0.14.0
grafana-infinity-ds 3.0.0
PostgreSQL Related
Patroni 4.0.5
PolarDB 15.12.3.0-e1e6d85b
IvorySQL 4.4
pgbackrest 2.54.2
pev2 1.14
WiltonDB 13.17
PostgreSQL Extensions
pgspider_ext 1.3.0 (new extension)
apache age 13–17 el rpm (1.5.0)
timescaledb 2.18.2 → 2.19.0
citus 13.0.1 → 13.0.2
documentdb 1.101-0 → 1.102-0
pg_analytics 0.3.4 → 0.3.7
pg_search 0.15.2 → 0.15.8
pg_ivm 1.9 → 1.10
emaj 4.4.0 → 4.6.0
pgsql_tweaks 0.10.0 → 0.11.0
pgvectorscale 0.4.0 → 0.6.0 (pgrx 0.12.5)
pg_session_jwt 0.1.2 → 0.2.0 (pgrx 0.12.6)
wrappers 0.4.4 → 0.4.5 (pgrx 0.12.9)
pg_parquet 0.2.0 → 0.3.1 (pgrx 0.13.1)
vchord 0.2.1 → 0.2.2 (pgrx 0.13.1)
pg_tle 1.2.0 → 1.5.0
supautils 2.5.0 → 2.6.0
sslutils 1.3 → 1.4
pg_profile 4.7 → 4.8
pg_snakeoil 1.3 → 1.4
pg_jsonschema 0.3.2 → 0.3.3
pg_incremental 1.1.1 → 1.2.0
pg_stat_monitor 2.1.0 → 2.1.1
ddl_historization 0.7 → 0.0.7 (bug fix)
pg_sqlog 3.1.7 → 1.6 (bug fix)
pg_random removed development suffix (bug fix)
asn1oid 1.5 → 1.6
table_log 0.6.1 → 0.6.4
Interface Changes
Added new Docker parameters: docker_data and docker_storage_driver (#521 by @waitingsong)
Added new Infra parameter: alertmanager_port, which lets you specify the AlertManager port
Added new Infra parameter: certbot_sign, apply for cert during nginx init? (false by default)
Added new Infra parameter: certbot_email, specifying the email used when requesting certificates via Certbot
Added new Infra parameter: certbot_options, specifying additional parameters for Certbot
Updated IvorySQL to place its default binary under /usr/ivory-4 starting in IvorySQL 4.4
Changed the default for pg_lc_ctype and other locale-related parameters from en_US.UTF-8 to C
For PostgreSQL 17, if using UTF8 encoding with C or C.UTF-8 locales, PostgreSQL’s built-in localization rules now take priority
configure automatically detects whether C.utf8 is supported by both the PG version and the environment, and adjusts locale-related options accordingly
Set the default IvorySQL binary path to /usr/ivory-4
Updated the default value of pg_packages to pgsql-main patroni pgbouncer pgbackrest pg_exporter pgbadger vip-manager
Updated the default value of repo_packages to [node-bootstrap, infra-package, infra-addons, node-package1, node-package2, pgsql-utility, extra-modules]
Removed LANG and LC_ALL environment variable settings from /etc/profile.d/node.sh
Now using bento/rockylinux-8 and bento/rockylinux-9 as the Vagrant box images for EL
Added a new alias, extra_modules, which includes additional optional modules
You can install PostgreSQL Kernels along with & 404 extensions with a simple command. Besides,
pig v0.3 is also embedded & shipped with the latest Pigsty v3.3.0.
New Features
pig build subcommand with the ability to set up extension building environment
pig build repo # init build repo (=repo set -ru)pig build tool # init build toolsetpig build rust # init rustc & pgrx (0.12.9)pig build spec # init rpm/deb spec repopig build get # get extension src tarballpig build ext # build extension## download big tarballpig build get std # download std small tarballpig build get all # download all source tarballpig build get pg_mooncake
pig build get pg_duckdb
pig build get omnigres
pig build get plv8
pig build get citus
pig build ext citus
pig build ext timescaledb
You can download the extension and its dependencies with import subcommand, activate different postgres major versions with link, and prepare building env with build subcommand
pig ext list [query]# list & search extensionpig ext info [ext...]# get information of a specific extensionpig ext status [-v]# show installed extension and pg statuspig ext add [ext...]# install extension for current pg versionpig ext rm [ext...]# remove extension for current pg versionpig ext update [ext...]# update extension to the latest versionpig ext import [ext...]# download extension to local repopig ext link [ext...]# link postgres installation to pathpig ext build [ext...]# setup building env for extension
Repo Management
You can now create a local repo and create a tarball (offline package) from it, copy it to somewhere (e.g., without internet access), and create a repo from that offline package:
pig repo list # available repo list (info)pig repo info [repo|module...]# show repo info (info)pig repo status # show current repo status (info)pig repo add [repo|module...]# add repo and modules (root)pig repo rm [repo|module...]# remove repo & modules (root)pig repo update # update repo pkg cache (root)pig repo create # create repo on current system (root)pig repo boot # boot repo from offline package (root)pig repo cache # cache repo as offline package (root)
Pigsty Management
The pig can also be used as a CLI tool for Pigsty — the battery-include free PostgreSQL RDS
pig sty init # install embed pigsty to ~/pigstypig sty boot # install ansible and other pre-depspig sty conf # auto-generate pigsty.yml config filepig sty install # run the install.yml playbook
Self-Updating
To update pig itself to the latest version, you can use the following command:
pig update
Info
Now pig info provides more details about your OS & PG environment:
$ pig info
# [Configuration] ================================
Pig Version : 0.1.0
Pig Config : /home/vagrant/.pig/config.yml
Log Level : info
Log Path : stderr
# [OS Environment] ===============================
OS Distro Code : el9
OS Architecture : amd64
OS Package Type : rpm
OS Vendor ID : rocky
OS Version : 9
OS Version Full : 9.3
OS Version Code : el9
# [PG Environment] ===============================
Installed:
* PostgreSQL 17.2 74 Extensions
Active:
PG Version : PostgreSQL 17.2
Config Path : /usr/pgsql-17/bin/pg_config
Binary Path : /usr/pgsql-17/bin
Library Path : /usr/pgsql-17/lib
Extension Path : /usr/pgsql-17/share/extension
# [Pigsty Environment] ===========================
Inventory Path : /home/vagrant/pigsty/pigsty.yml
Pigsty Home : /home/vagrant/pigsty
Embedded Version : 3.2.0
# [Network Conditions] ===========================
pigsty.cc ping ok: 141 ms
pigsty.io ping ok: 930 ms
google.com request error
Internet Access : true
Pigsty Repo : pigsty.io
Inferred Region : china
Latest Pigsty Ver : v3.2.0
The pig util is a standalone go binary with no dependencies. you can just download the binary or use the following commands to add the repo and install it via package manager (recommended).
For Ubuntu 22.04 / 24.04 & Debian 12 or any compatible platforms:
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [trusted=yes] https://repo.pigsty.io/apt/infra generic main
EOFsudo apt update; sudo apt install -y pig
For EL 8/9 and compatible platforms:
sudo tee /etc/yum.repos.d/pigsty.repo > /dev/null <<-'EOF'
[pigsty-infra]
name=Pigsty Infra for $basearch
baseurl=https://repo.pigsty.io/yum/infra/$basearch
enabled = 1
gpgcheck = 0
module_hotfixes=1
EOFsudo yum makecache; sudo yum install -y pig
For mainland china user: consider replace the repo.pigsty.io with repo.pigsty.cc
Compatibility
pig runs on: RHEL 8/9, Ubuntu 22.04/24.04, and Debian 12, on both amd64/arm64 arch
Code
Distribution
x86_64
aarch64
el9
RHEL 9 / Rocky9 / Alma9 / …
PG 17 - 13
PG 17 - 13
el8
RHEL 8 / Rocky8 / Alma8 / …
PG 17 - 13
PG 17 - 13
u24
Ubuntu 24.04 (noble)
PG 17 - 13
PG 17 - 13
u22
Ubuntu 22.04 (jammy)
PG 17 - 13
PG 17 - 13
d12
Debian 12 (bookworm)
PG 17 - 13
PG 17 - 13
Here are some bad cases and limitations for the above distros:
citus is not available on aarch64 and ubuntu 24.04
PG Exporter brings ultimate monitoring experience to your PostgreSQL with declarative config, dynamic planning, and customizable collectors.
It provides 600+ metrics and ~3K time series per instance, covers everything you’ll need for PostgreSQL observability.
Check the GitHub Repo for more details.
The latest stable version of pg_exporter is v1.0.3
Remove the monitor schema requirement for pg_query collectors (you have to ensure it with search_path or just
install pg_stat_statements in the default public schema)
Fix pgbouncer version parsing message level from info to debug
curl https://repo.pigsty.io/pig | bash # download and install the pig CLI toolpig repo add all pigsty -u # add pigsty-pgsql repo and update cache
apt
# Add Pigsty's GPG public key to your system keychain to verify package signaturescurl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg
# Get Debian distribution codename (distro_codename=jammy, focal, bullseye, bookworm), and write the corresponding upstream repository address to the APT List filedistro_codename=$(lsb_release -cs)sudo tee /etc/apt/sources.list.d/pigsty-io.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql/${distro_codename} ${distro_codename} main
EOF# Refresh APT repository cachesudo apt update
2025-11-20
Name
Old
New
Comment
vchord
0.5.3
1.0.0
pg_later
0.3.1
0.4.0
pgvectorscale
0.8.0
0.9.0
-pg13, +pg18
pglite_fusion
0.0.5
0.0.6
pgx_ulid
0.2.1
0.2.2
pg_search
0.19.5
0.19.7
resume PIGSTY building
citus
13.2.0
13.2.0
official tag
timescaledb
2.23.0
2.23.1
pg_profile
4.10
4.11
pglinter
1.0.0
new
pg_typeid
0.3.0
head with pg18 support
pg_enigma
0.4.0
vonng patched pgrx version
pg_retry
1.0.0
new, pg17-18
pg_biscuit
1.0
new, pg16-18
pg_weighted_statistics
1.0.0
new, pg13-18
documentdb
0.106
0.107
ferretdb fork
PolarDB
15.15
15.15.5.0-38948055
2025-11-10
Add PostgreSQL 18 support for almost all extensions
curl https://repo.pigsty.io/pig | bash # download and install the pig CLI toolpig repo add infra -u # add pigsty-pgsql repo and update cache
2025-11-20
Name
Old
New
Comment
pgschema
-
1.4.2
new
pgflo
-
0.0.15
new
vector
0.51.0
0.51.1
bugfix release
sealos
5.0.1
5.1.1
etcd
3.6.5
3.6.6
duckdb
1.4.1
1.4.2
pg_exporter
1.0.2
1.0.3
pig
0.7.1
0.7.2
grafana
12.1.0
12.3.0
pg_timetable
6.1.0
6.2.0
genai-toolbox
0.16.0
0.21.0
timescaledb-tools
0.18.0
0.18.1
move to infra
timescaledb-event-streamer
0.12.0
0.20.0
tigerbeetle
0.16.60
0.16.65
victoria-metrics
1.129.1
1.130.0
victorialogs
1.37.2
1.38.0
grafana-victorialogs-ds
0.21.4
0.22.1
grafana-victoriametrics-ds
0.19.6
0.19.7
grafana-plugins
12.0.0
12.3.0
2025-11-11
Name
Old
New
Comment
grafana
12.1.0
12.2.1
download address changed
prometheus
3.6.0
3.7.3
pushgateway
1.11.1
1.11.2
alertmanager
0.28.1
0.29.0
nginx_exporter
1.5.0
1.5.1
node_exporter
1.9.1
1.10.2
pgbackrest_exporter
0.20.0
0.21.0
redis_exporter
1.77.0
1.80.0
duckdb
1.4.0
1.4.1
dblab
0.33.0
0.34.2
pg_timetable
5.13.0
6.1.0
vector
0.50.0
0.51.0
rclone
1.71.1
1.71.2
victoria-metrics
1.126.0
1.129.1
victoria-logs
1.35.0
1.37.2
grafana-victorialogs-ds
0.21.0
0.21.4
grafana-victoriametrics-ds
0.19.4
0.19.6
grafana-infinity-ds
3.5.0
3.6.0
genai-toolbox
0.16.0
0.18.0
pev2
1.16.0
1.17.0
pig
0.6.2
0.7.1
2025-10-18
Name
Old
New
Comment
prometheus
3.5.0
3.6.0
nginx_exporter
1.4.2
1.5.0
mysqld_exporter
0.17.2
0.18.0
redis_exporter
1.75.0
1.77.0
mongodb_exporter
0.47.0
0.47.1
victoria-metrics
1.121.0
1.126.0
vicotira-logs
1.25.1
1.35.0
duckdb
1.3.2
1.4.0
etcd
3.6.4
3.6.5
restic
0.18.0
0.18.1
tigerbeetle
0.16.54
0.16.60
grafana-victorialogs-ds
0.19.3
0.21.0
grafana-victoriametrics-ds
0.18.3
0.19.4
grafana-infinity-ds
3.3.0
3.5.0
genai-toolbox
0.9.0
0.16.0
grafana
12.1.0
12.2.0
vector
0.49.0
0.50.0
rclone
1.70.3
1.71.1
minio
20250723155402
20250907161309
mcli
20250721052808
20250813083541
2025-08-15
Name
Old
New
Comment
grafana
12.0.0
12.1.0
pg_exporter
1.0.1
1.0.2
pig
0.6.0
0.6.1
vector
0.48.0
0.49.0
redis_exporter
1.74.0
1.75.0
mongo_exporter
0.46.0
0.47.0
victoriametrics
1.121.0
1.123.0
victorialogs:
1.25.0
1.28.0
grafana-victoriametrics-ds
0.17.0
0.18.3
grafana-victorialogs-ds
0.18.3
0.19.3
grafana-infinity-ds
3.3.0
3.4.1
etcd
3.6.1
3.6.4
ferretdb
2.3.1
2.5.0
tigerbeetle
0.16.50
0.16.54
genai-toolbox
0.9.0
0.12.0
2025-07-24
Name
Old
New
Comment
FerretDB
-
2.4.0
work with documentdb 1.105
etcd
-
3.6.3
minio
-
20250723155402
mcli
-
20250721052808
ivorysql
-
4.5-0ffca11-20250709
fix libxcrypt deps issue
2025-07-16
Name
Old
New
Comment
genai-toolbox
0.8.0
0.9.0
MCP toolbox for various DBMS
victoriametrics
1.120.0
1.121.0
split into various packages
victorialogs
1.24.0
1.25.0
split into various packages
prometheus
3.4.2
3.5.0
duckdb
1.3.1
1.3.2
etcd
3.6.1
3.6.2
tigerbeetle
0.16.48
0.16.50
grafana-victoriametrics-ds
0.16.0
0.17.0
rclone
1.69.3
1.70.3
pig
0.5.0
0.6.0
pev2
1.15.0
1.16.0
pg_exporter
1.0.0
1.0.1
2025-07-04
Name
Old
New
Comment
prometheus
3.4.1
3.4.2
-
grafana
12.0.1
12.0.2
-
vector
0.47.0
0.48.0
-
rclone
1.69.0
1.70.2
-
vip-manager
3.0.0
4.0.0
-
blackbox_exporter
0.26.0
0.27.0
-
redis_exporter
1.72.1
1.74.0
-
duckdb
1.3.0
1.3.1
-
etcd
3.6.0
3.6.1
-
ferretdb
2.2.0
2.3.1
-
dblab
0.32.0
0.33.0
-
tigerbettle
0.16.41
0.16.48
-
grafana-victorialogs-ds
0.16.3
0.18.1
-
grafana-victoriametrics-ds
0.15.1
0.16.0
-
grafana-inifinity-ds
3.2.1
3.3.0
-
victorialogs
1.22.2
1.24.0
-
victoriametrics
1.117.1
1.120.0
-
2025-06-01
Name
Old
New
Comment
grafana
-
12.0.1
-
prometheus
-
3.4.1
-
keepalived_exporter
-
1.7.0
-
redis_exporter
-
1.73.0
-
victoriametrics
-
1.118.0
-
victorialogs
-
1.23.1
-
tigerbeetle
-
0.16.42
-
grafana-victorialogs-ds
-
0.17.0
-
grafana-infinity-ds
-
3.2.2
-
2025-05-22
Name
Old
New
Comment
dblab
-
0.32.0
-
prometheus
-
3.4.0
-
duckdb
-
1.3.0
-
etcd
-
3.6.0
-
pg_exporter
-
1.0.0
-
ferretdb
-
2.2.0
-
rclone
-
1.69.3
-
minio
-
20250422221226
The last version with admin GUI
mcli
-
20250416181326
-
nginx_exporter
-
1.4.2
-
keepalived_exporter
-
1.6.2
-
pgbackrest_exporter
-
0.20.0
-
redis_exporter
-
1.27.1
-
victoriametrics
-
1.117.1
-
victorialogs
-
1.22.2
-
pg_timetable
-
5.13.0
-
tigerbeetle
-
0.16.41
-
pev2
-
1.15.0
-
grafana
-
12.0.0
-
grafana-victorialogs-ds
-
0.16.3
-
grafana-victoriametrics-ds
-
0.15.1
-
grafana-infinity-ds
-
3.2.1
-
grafana_plugins
-
12.0.0
-
2025-04-23
Name
Old
New
Comment
mtail
-
3.0.8
new
pig
-
0.4.0
-
pg_exporter
-
0.9.0
-
prometheus
-
3.3.0
-
pushgateway
-
1.11.1
-
keepalived_exporter
-
1.6.0
-
redis_exporter
-
1.70.0
-
victoriametrics
-
1.115.0
-
victoria_logs
-
1.20.0
-
duckdb
-
1.2.2
-
pg_timetable
-
5.12.0
-
vector
-
0.46.1
-
minio
-
20250422221226
-
mcli
-
20250416181326
-
2025-04-05
Name
Old
New
Comment
pig
-
0.3.4
-
etcd
-
3.5.21
-
restic
-
0.18.0
-
ferretdb
-
2.1.0
-
tigerbeetle
-
0.16.34
-
pg_exporter
-
0.8.1
-
node_exporter
-
1.9.1
-
grafana
-
11.6.0
-
zfs_exporter
-
3.8.1
-
mongodb_exporter
-
0.44.0
-
victoriametrics
-
1.114.0
-
minio
-
20250403145628
-
mcli
-
20250403170756
-
2025-03-23
Name
Old
New
Comment
etcd
-
3.5.20
-
pgbackrest_exporter
-
0.19.0
rebuild
victorialogs
-
1.17.0
-
vslogcli
-
1.17.0
-
2025-03-17
Name
Old
New
Comment
kafka
-
4.0.0
-
Prometheus
-
3.2.1
-
AlertManager
-
0.28.1
-
blackbox_exporter
-
0.26.0
-
node_exporter
-
1.9.0
-
mysqld_exporter
-
0.17.2
-
kafka_exporter
-
1.9.0
-
redis_exporter
-
1.69.0
-
DuckDB
-
1.2.1
-
etcd
-
3.5.19
-
FerretDB
-
2.0.0
-
tigerbeetle
-
0.16.31
-
vector
-
0.45.0
-
VictoriaMetrics
-
1.114.0
-
VictoriaLogs
-
1.16.0
-
rclone
-
1.69.1
-
pev2
-
1.14.0
-
grafana-victorialogs-ds
-
0.16.0
-
grafana-victoriametrics-ds
-
0.14.0
-
grafana-infinity-ds
-
3.0.0
-
timescaledb-event-streamer
-
0.12.0
new
restic
-
0.17.3
new
juicefs
-
1.2.3
new
2025-02-12
Name
Old
New
Comment
pushgateway
1.10.0
1.11.0
-
alertmanager
0.27.0
0.28.0
-
nginx_exporter
1.4.0
1.4.1
-
pgbackrest_exporter
0.18.0
0.19.0
-
redis_exporter
1.66.0
1.67.0
-
mongodb_exporter
0.43.0
0.43.1
-
VictoriaMetrics
1.107.0
1.111.0
-
VictoriaLogs
v1.3.2
1.9.1
-
DuckDB
1.1.3
1.2.0
-
Etcd
3.5.17
3.5.18
-
pg_timetable
5.10.0
5.11.0
-
FerretDB
1.24.0
2.0.0
-
tigerbeetle
0.16.13
0.16.27
-
grafana
11.4.0
11.5.1
-
vector
0.43.1
0.44.0
-
minio
20241218131544
20250207232109
-
mcli
20241121172154
20250208191421
-
rclone
1.68.2
1.69.0
-
2024-11-19
Name
Old
New
Comment
Prometheus
2.54.0
3.0.0
-
VictoriaMetrics
1.102.1
1.106.1
-
VictoriaLogs
v0.28.0
1.0.0
-
MySQL Exporter
0.15.1
0.16.0
-
Redis Exporter
1.62.0
1.66.0
-
MongoDB Exporter
0.41.2
0.42.0
-
Keepalived Exporter
1.3.3
1.4.0
-
DuckDB
1.1.2
1.1.3
-
etcd
3.5.16
3.5.17
-
tigerbeetle
16.8
0.16.13
-
grafana
-
11.3.0
-
vector
-
0.42.0
-
10 - Service
Expert Consultation and Subscription Plans
Pigsty aims to consolidate the strengths of the PostgreSQL ecosystem and replace manual database operations with auto-pilot software.
It’s completely open-source and FREE, with absolutely no warranty.
While Professional Service is important in enterprise environments, we offer expert consultation and subscription plans tailored to organizations seeking expertise and dedicated assistance.
Pigsty Subscription
Long-Term subscription plans for Pigsty users
Expert Consulting
On-Demand expert consultation and one-time support
On-Demand Expert
On-Demand Expert Consultation
We (I) have on-demand expert services, 400 $/hour, can be used for:
configure pigsty according to your business needs
self-hosting Supabase, odoo, dify, gitlab, etc…
consulting about database, cloud, observability, and related topics
look into failures and optimize slow queries (no on-call)
deliver offline install package for specific OS and PG (in list)
On-Demand expert consultation services DO NOT guarantee any response time or availability.
If you need long-term, reliable support with SLA commitment, consider our subscription plans.
Long-Term Subscription Plans
Long-Term expert support for Pigsty and PostgreSQL
Best practices guidance for production services
Immediate response to critical failures with SLA
Post-mortem analysis to identify root causes of failures
Performance optimization and dashboard interpretation
Database architecture that meets compliance requirements
Migration assistance from other databases to PostgreSQL
Self-hosting PG-related stuff like Supabase, Odoo, Gitlab,…
Running pigsty on legacy OS such as el7, debian 11, ubuntu 22, …
Launch on-perm observability stack with Prometheus / Grafana
Building & Delivering customize extensions on demand
Running alternative PG kernel forks instead of vanilla PG
Written commitment for AGPLv3 license liability exemptions
Distribute Pigsty as DBaaS or OEM it as a commercial product
In addition to the open-source version, Pigsty offers 3 subscription plans: Standard, Professional and Enterprise.
You can choose the appropriate subscription plan based on your actual situation and needs.
The number of nodes is defined as the total number of independent IP addresses that exist as Hosts in the config inventory of a Pigsty deployment, i.e., the total number of nodes managed by Pigsty.
Item / Plan
OSS
Standard
Professional
Enterprise
License
AGPLv3
Commercial
Commercial
Commercial
Node Limit
unlimited
≤ 5
≤ 15
≤ 50, up to unlimited
Warranty
No Warranty
+Pigsty
+Kernel
+Extension
Pigsty Bug Fix
latest minor
latest major
all
all
PG Support
17
17,16
17,16,15,14,13
back to 9.x+
PG Extension
+Debug Symbol
+All Versions
+Customize Build
OS Support
EL9, D12, U22
+ARM64
+EL8, U24
+EL7,D11,U20 Bespoke
Arch Support
x86_64
x86_64, ARM64
x86_64, ARM64
x86_64, ARM64
Offline Package
el9,d12,u22
+ARM64
+OS Minor
Bespoke
Extra Modules
+ARM64
+Pro Modules
+Pilot Modules
Advanced CLI
yes
yes
yes
DBA Hours
one-time setup
up to 5h/mo
up to 10h/mo
Expert Support
available
1 man·day/yr
2 man·day/yr
Support Grade
standard
professional
enterprise
Support SLA
5x8, same day
5x8, < 4h
7x24, < 30min
Price
Free
8,000 $/year
24,000 $/year
60,000 $/year
Pigsty OSS
Free under the AGPLv3 License, No Warranty
Pigsty is built on open source and also gives back to the open source community. It is a gift to the PostgreSQL community and all users —
you can get the complete core functionality of Pigsty without any payment. Of course, as is typical with open-source software,
the Pigsty Open Source Edition does not offer any warranty service and bears no responsibility for any consequences arising from its use.
If you require a warranty, please consider our subscription services.
The Pigsty Open Source Edition is released under the AGPLv3 license, which is a copyleft, strict open-source license.
If you are an ordinary end user (i.e., users other than public cloud vendors or database vendors), we will not pursue any action against your secondary development of Pigsty.
In practice, it is effectively licensed under the more permissive Apache 2.0 license for most end users.
If you discover any defects in Pigsty, we highly encourage you to submit an Issue on GitHub to help us improve.
If you have any questions, you can seek help in the Community.
For the open-source version, we provide prebuilt standard offline packages for PostgreSQL 17 on three precisely targeted OS distributions —
EL 9.6, Debian 12.11, Ubuntu 24.04.2 — with the latest minor versions (as a form of open-source support, aarch64 offline packages are also provided for Debian 12).
By using the Pigsty Open Source Edition, entry-level developers and DevOps engineers can access 70%+ of the capabilities of a professional DBA.
Even without a dedicated database expert, you can easily set up a high-availability, high-performance, easy-to-maintain, secure, and reliable PostgreSQL database cluster.
If self-hosted in the cloud, you can immediately save on the price difference between EC2/ESSD and RDS services, achieving significant cost reductions up to 90+%.
Code
Distro Major
Minor
x86_64
aarch64
EL9
RHEL 9 / Rocky9 / Alma9
9.6
1716151413
1716151413
D12
Debian 12 (bookworm)
12.11
1716151413
1716151413
U24
Ubuntu 24.04 (noble)
24.04.2
1716151413
1716151413
Pigsty Standard
Economical choice for SMBs, startups, and freelancers
The Pigsty Standard subscription provides an affordable safety net for small and medium businesses—we offer warranty and support coverage for the Pigsty software itself.
The Standard subscription uses a dedicated commercial license, delivering a written contractual commitment that waives the AGPLv3 derived work open-source obligations of Pigsty.
We provide a one-time architectural consultation service to Standard subscription customers. Based on your environment and available resources,
we will propose a suitable database architecture design. Whether you want to use PostgreSQL to build a business system or self-host Odoo, Dify, Supabase, Gitlab,
or other applications, we can provide comprehensive support, including offline installation and network solutions.
The Pigsty Standard subscription includes basic expert ticketing and Q&A services. We commit to responding to your questions within working hours.
For more complex issues requiring additional support, our expert support (man-day) service is also available for purchase.
Our PostgreSQL expertise can help you avoid numerous pitfalls, saving you time, effort, and costs.
For mainstream open-source Linux distributions (EL9, Debian 12, Ubuntu 22.04) at their latest stable minor versions,
the Pigsty Standard subscription provides offline software installation packages for both x86_64 and aarch64.
These packages include the PostgreSQL 17 kernel and all available extensions, tested to ensure quick, stable,
and efficient installation with consistent versions, independent of network environment and upstream repository changes.
The starting price for Pigsty Standard is 8,000 $ / year, roughly equivalent to the annual fee for 4 vCPUs of AWS HA RDS PG or the salary of an intern with a monthly wage of $600.
Code
Distro Major
Minor
x86_64
aarch64
EL9
RHEL 9 / Rocky9 / Alma9
9.6
1716151413
1716151413
D12
Debian 12 (bookworm)
12.11
1716151413
1716151413
U24
Ubuntu 24.04 (noble)
24.04.2
1716151413
1716151413
Pigsty Professional
Suitable choice for typical enterprise users
The Pigsty Professional subscription builds upon the Standard offering with more advanced consultation services.
The Professional subscription includes analysis of complex issues and performance bottleneck optimization, ensuring you can access top-level DBA expertise at critical moments.
We provide comprehensive architectural consultation for Professional subscription customers.
Based on your business needs and resource availability, we develop the optimal database architecture design and ensure its successful implementation.
We also help with high availability testing and PITR exercises, and provide training on Pigsty’s monitoring system, configuration methods, and management commands.
The Pigsty Professional subscription offers enhanced support. We provide one expert man-day per year,
along with monthly DBA consultation and Q&A not exceeding five hours. We also offer a faster SLA response time:
for routine questions, we guarantee a response within four hours during weekday working hours (5x8).
The Pigsty Professional subscription supports a broader range of operating systems, adding EL 8 and Ubuntu 24.04 to the list of supported distributions,
and provides aarch64 support for all of these versions. If you are not using the latest minor version, we can customize offline software packages for your specified minor version.
Moreover, the offline packages include all Pigsty feature modules, such as PG branch kernels (IvorySQL, PolarDB, Babelfish) and all Pro/Beta modules.
Pigsty Professional provides support for the three most recent major PostgreSQL releases (17, 16, 15),
and offers expert guidance for PostgreSQL major version upgrades as well as Pigsty upgrades.
The starting price for Pigsty Professional is 24,000 $ / year, roughly equivalent to the annual fee for 11 vCPUs of AWS HA RDS PG, or the annual salary of a junior DevOps engineer with a monthly wage of $2,000.
Code
Distro Major
Minor
x86_64
aarch64
EL9
RHEL 9 / Rocky9 / Alma9
9.x
1716151413
1716151413
D12
Debian 12 (bookworm)
12.x
1716151413
1716151413
U22
Ubuntu 22.04 (jammy)
22.04.x
1716151413
1716151413
U24
Ubuntu 24.04 (noble)
24.04.x
1716151413
1716151413
EL8
RHEL 8 / Rocky8 / Alma8
8.x
1716151413
1716151413
Pigsty Enterprise
Designed for mission-critical scenarios
Pigsty Enterprise is designed for medium and large enterprises or mission-critical scenarios requiring strict SLAs.
With the Enterprise subscription, we offer the highest level of support to meet all your database needs.
Under the Enterprise subscription, we help you design and implement the optimal database architecture solution.
Beyond database drills, stress tests, and performance evaluations, we also provide consultancy and training on management systems,
helping you build a comprehensive database management framework that meets various security and compliance requirements.
Pigsty Enterprise includes two expert man-days per year, plus a monthly DBA consultation and Q&A of up to 10 hours.
For routine issues, we guarantee a response within 30 minutes, 7x24, and always prioritize your requests.
The Pigsty Enterprise subscription offers the widest range of OS support, adding EL7, Debian 11, and Ubuntu 20.04, including EOL releases.
We can also customize support for Euler, Anolis, UOS, Kylin, TencentOS, AliOS, OpenCloudOS, and other Linux distributions.
The Pigsty Enterprise subscription covers all PostgreSQL major releases (13–17) within their active lifecycles,
ensuring smooth in-place upgrades across different major versions. Using the included man-days,
you can migrate your PostgreSQL clusters to the latest major version via a zero-downtime, blue-green deployment process.
Pigsty Enterprise permits usage of Pigsty as DBaaS at a specified scale, allowing you to build and sell cloud database services.
It also allows OEM use—you may distribute Pigsty with your own logo, trademarks, and branding within the agreed scope.
The starting price for Pigsty Enterprise is 60,000 $ / year, which is equivalent to 27 vCPUs of AWS RDS for PostgreSQL, or a developer with a monthly salary of $5,000.
Code
Distro Major
Minor
x86_64
aarch64
EL9
RHEL 9 / Rocky9 / Alma9
9.x
1716151413
1716151413
D12
Debian 12 (bookworm)
12.x
1716151413
1716151413
U22
Ubuntu 22.04 (jammy)
22.04.x
1716151413
1716151413
U24
Ubuntu 24.04 (noble)
24.04.x
1716151413
1716151413
EL8
RHEL 8 … / Anolis8
8.x
1716151413
1716151413
EL7
RHEL 7 … / UOS / Euler
7.x
1716151413
1716151413
D11
Debian 11 (bullseye)
11.x
1716151413
1716151413
U20
Ubuntu 20.04 (focal)
20.04.x
1716151413
1716151413
Pricing
Pigsty subscriptions are annual, beginning on the agreed date. Payment before expiration implies automatic renewal.
Continuous subscriptions receive discounts: 5% off the second year, 10% off subsequent years, and 15% off for three-year commitments.
After subscription expiration, non-renewal results in cessation of updates, technical support, and consulting,
though previously installed Pro software remains usable. Non-renewal gaps do not require back payment upon re-subscription, but loyalty discounts are reset.
Pigsty’s pricing delivers exceptional value—providing immediate access to top-tier DBA expertise and database management practices
at a cost that compares favorably to hiring full-time experts or using cloud database services. For reference, market prices for comparable enterprise database services include:
AWS RDS for PostgreSQL HA: $160 – $220 / (vCPU·month), equivalent to $1,920 ~ $2,640/year (per vCPU).
pg-test-primary: Read-Write Service (route to primary pgbouncer)
pg-test-replica: Read-Only Service (route to replicas pgbouncer)
pg-test-default: Direct RW Service (route to primary postgres)
pg-test-offline: Offline Read Service (route to dedicated postgres)
HA Description
The PostgreSQL cluster is managed by Patroni,
which is a battle-tested HA solution for PostgreSQL.
It will set up PG Replication on multiple nodes, and perform automatic failover when the primary node is down.
The backup is handled by pgBackRest, which is a powerful backup tool for PostgreSQL,
which supports incremental backup/restore, compression, encryption, backup to local disk or S3 / MinIO.
The pgbouncer is a lightweight connection pooler which can increase the performance with high-concurrency.
It is 1:1 deployed with the Postgres server and used by primary / replica services by default.
The services are exposed by HAProxy, which is a high-performance TCP/HTTP load balancer, it’s part of NODE module.
And 4 default services are auto exposed in an idempotent way on all cluster nodes.
The application can visit any of the haproxy to access the Postgres cluster, and the traffic will be routed to the correct instance based on patroni health check endpoints.
So failover is transparent to the apps.
The patroni requires a functioning ETCD in your deployment, and pgbackrest can use the optional MinIO as centralized backup storage;
and monitoring exporters will collect metrics & logs into the Infra module.
Components
The PGSQL Node consists of the following components (some can be disabled)
Component
Port
Description
postgres
5432
PostgreSQL Server Process Managed by Patroni
pgbouncer
6432
Pgbouncer Connection Pool
pgbackrest
-
Backup and point-in-time-recovery tools
patroni
8008
Patroni HA Component, Manage postgres
primary @ haproxy
5433
Primary connection pool: Read/Write Service
replica @ haproxy
5434
Replica connection pool: Read-only Service
default @ haproxy
5436
Primary Direct Connect Service
offline @ haproxy
5438
Offline Direct Connect: Offline Read Service
pg_exporter
9630
postgres Monitoring Metrics Exporter
pgbouncer_exporter
9631
pgbouncer Monitoring Metrics Exporter
pgbackrest_exporter
9854
pgbackrest Monitoring Metrics Exporter
vip-manager
-
Bind VIP to the primary
Interaction
Meanwhile, the Infra Node consists of the following components which interact with PGSQL.
Component
Port
Domain
Description
nginx
80
h.pigsty
Web Service Portal (YUM/APT Repo)
alertmanager
9059
a.pigsty
Alert Aggregation and delivery
prometheus
9058
p.pigsty
Monitoring Time Series Database
grafana
3000
g.pigsty
Visualization Platform
lok
3100
-
Logging Collection Server
pushgateway
9091
-
Collect One-Time Job Metrics
blackbox_exporter
9115
-
Blackbox Probing
dnsmasq
53
-
DNS Server
chronyd
123
-
NTP Time Server
ansible
-
-
Run playbooks
Cluster DNS is resolved by DNSMASQ on infra nodes
Cluster VIP is managed by vip-manager, which binds to cluster primary.
vip-manager will acquire cluster leader info written by patroni from etcd cluster directly
Cluster services are exposed by Haproxy on nodes, services are distinguished by node ports (543x).
Haproxy port 9101: monitoring metrics & stats & admin page
Haproxy port 5433: default service that routes to primary pgbouncer: primary
Haproxy port 5434: default service that routes to replica pgbouncer: replica
Haproxy port 5436: default service that routes to primary postgres: default
Haproxy port 5438: default service that routes to offline postgres: offline
HAProxy will route traffic based on health check information provided by patroni.
Pgbouncer is a connection pool that listens to port 6432
1:1 Deployed with the Postgres server through a local unix socket.
Production traffic (Primary/Replica) will go through pgbouncer by default
Bypass pgbouncer for primary/replica services by setting pg_default_service_dest ) to postgres
Default/Offline service will always bypass pgbouncer and connect to target Postgres directly.
Postgres provides relational database services @ port 5432
Install PGSQL module on multiple nodes will automatically form a HA cluster based on replication.
PostgreSQL is supervised by patroni by default.
Patroni will supervise PostgreSQL server @ port 8008 by default
Patroni spawn postgres servers as the child process
Patroni uses etcd as DCS: config storage, failure detection, and leader election.
Patroni will provide Postgres information through a health check, used by HAProxy
Patroni metrics will be scraped by prometheus on infra nodes
PG Exporter will expose postgres metrics @ port 9630
Pgbouncer Exporter will expose pgbouncer metrics @ port 9631
Pgbouncer’s metrics will be scraped by prometheus on infra nodes
pgBackRest will work on the local repo by default (pgbackrest_method)
If local (default) is used as the backup repo, primary’s pg_fs_backup is used as local backup repo
If minio is used, pgBackRest will create the repo on the dedicated MinIO cluster
Postgres-related logs (postgres,pgbouncer,patroni,pgbackrest) are exposed by promtail @ port 9080
Promtail will send logs to Loki on infra node
Full ER Diagram
There is one config inventory file and one infra corresponding to a Pigsty deployment.
And there may have multiple database clusters in a Pigsty deployment.
A Cluster / Instance may have multiple Databases, and Databases contain Tables and other Objects (Query, Index, Function, Seq, …).
11.2 - Configure
Describe and configure PostgreSQL clusters
You can define different types of instances & clusters.
Pigsty has PG 13 - 18 support. Lower major version (12-) “may” work, with no guarantee.
For legacy PG version support, consider our professional services.
To use a different major version, configure the pg_version variable.
Which can be globally configure with -v <ver> option.
No further changed needed as long as they are available in local / upstream repo.
Offline instance works like common replica instances, but it is used as a backup server in pg-test-replica service. That is to say, offline and primary instances serve only when all replica instances are down.
Pigsty uses asynchronous stream replication by default, which may have a small replication lag (10KB / 10ms). A small window of data loss may occur when the primary fails (can be controlled with pg_rpo), but it is acceptable for most scenarios.
But in some critical scenarios (e.g., financial transactions), data loss is totally unacceptable or read-your-write consistency is required. In this case, you can enable synchronous commit to ensure that.
To enable sync standby mode, you can simply use crit.yml template in pg_conf
To enable sync standby on existing clusters, config the cluster and enable synchronous_mode:
$ pg edit-config pg-test # run on admin node with admin user+++
-synchronous_mode: false# <--- old value+synchronous_mode: true# <--- new value synchronous_mode_strict: falseApply these changes? [y/N]: y
If synchronous_mode: true, the synchronous_standby_names parameter will be managed by patroni. It will choose a sync standby from all available replicas and write its name to the primary’s configuration file.
Quorum Commit
When sync standby is enabled, PostgreSQL will pick one replica as the standby instance, and all other replicas as candidates. Primary will wait until the standby instance flushes to disk before a commit is confirmed, and the standby instance will always have the latest data without any lags.
However, you can achieve an even higher/lower consistency level with the quorum commit (trade-off with availability).
For example, to have all 2 replicas to confirm a commit:
synchronous_mode:true# make sure synchronous mode is enabledsynchronous_node_count:2# at least 2 nodes to confirm a commit
If you have more replicas and wish to have more sync standby, increase synchronous_node_count accordingly. Beware of adjust synchronous_node_count accordingly when you append or remove replicas.
The postgres synchronous_standby_names parameter will be managed by patroni:
The classic quorum commit is to use majority of replicas to confirm a commit.
synchronous_mode:quorum # use quorum commitpostgresql:parameters:# change the PostgreSQL parameter `synchronous_standby_names`, use the `ANY n ()` notionsynchronous_standby_names:'ANY 1 (*)'# you can specify a list of standby names, or use `*` to match them all
Standby Cluster
You can clone an existing cluster and create a standby cluster, which can be used for migration, horizontal split, multi-az deployment, or disaster recovery.
A standby cluster’s definition is just the same as any other normal cluster, except there’s a pg_upstream defined on the primary instance.
For example, you have a pg-test cluster, to create a standby cluster pg-test2, the inventory may look like this:
# pg-test is the original clusterpg-test:hosts:10.10.10.11:{pg_seq:1, pg_role:primary }vars:{pg_cluster:pg-test }# pg-test2 is a standby cluster of pg-test.pg-test2:hosts:10.10.10.12:{pg_seq:1, pg_role:primary , pg_upstream:10.10.10.11}# <--- pg_upstream is defined here10.10.10.13:{pg_seq:2, pg_role:replica }vars:{pg_cluster:pg-test2 }
And pg-test2-1, the primary of pg-test2 will be a replica of pg-test and serve as a Standby Leader in pg-test2.
Just make sure that the pg_upstream parameter is configured on the primary of the backup cluster to pull backups from the original upstream automatically.
bin/pgsql-add pg-test # Creating the original clusterbin/pgsql-add pg-test2 # Creating a Backup Cluster
Delayed Cluster
A delayed cluster is a special type of standby cluster, which is used to recover “drop-by-accident” ASAP.
For example, if you wish to have a cluster pg-testdelay which has the same data as 1-day ago pg-test cluster:
# pg-test is the original clusterpg-test:hosts:10.10.10.11:{pg_seq:1, pg_role:primary }vars:{pg_cluster:pg-test }# pg-testdelay is a delayed cluster of pg-test.pg-testdelay:hosts:10.10.10.12:{pg_seq:1, pg_role:primary , pg_upstream:10.10.10.11, pg_delay:1d }10.10.10.13:{pg_seq:2, pg_role:replica }vars:{pg_cluster:pg-test2 }
patroni_primary_db has to be defined to specify the database to be managed
pg_dbsu_password has to be set to a non-empty string plain password if you want to use the pg_dbsupostgres rather than default pg_admin_username to perform admin commands
Besides, extra hba rules that allow ssl access from local & other data nodes are required. Which may looks like this
all:children:pg-citus0:# citus data node 0hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:{pg_cluster:pg-citus0 , pg_group:0}pg-citus1:# citus data node 1hosts:{10.10.10.11:{pg_seq:1, pg_role:primary } }vars:{pg_cluster:pg-citus1 , pg_group:1}pg-citus2:# citus data node 2hosts:{10.10.10.12:{pg_seq:1, pg_role:primary } }vars:{pg_cluster:pg-citus2 , pg_group:2}pg-citus3:# citus data node 3, with an extra replicahosts:10.10.10.13:{pg_seq:1, pg_role:primary }10.10.10.14:{pg_seq:2, pg_role:replica }vars:{pg_cluster:pg-citus3 , pg_group:3}vars:# global parameters for all citus clusterspg_mode:citus # pgsql cluster mode: cituspg_shard:pg-citus # citus shard name: pg-cituspatroni_citus_db:meta # citus distributed database namepg_dbsu_password:DBUser.Postgres# all dbsu password access for citus clusterpg_users:[{name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin ] } ]pg_databases:[{name:meta ,extensions:[{name:citus }, { name: postgis }, { name: timescaledb } ] } ]pg_hba_rules:- {user:'all' ,db:all ,addr:127.0.0.1/32 ,auth:ssl ,title:'all user ssl access from localhost'}- {user:'all' ,db:all ,addr:intra ,auth:ssl ,title:'all user ssl access from intranet'}
And you can create distributed table & reference table on the coordinator node. Any data node can be used as the coordinator node since citus 11.2.
If pg_mode is set to citus or gpsql, pg_shard and pg_group will be required for horizontal sharding clusters.
pg_cluster
name: pg_cluster, type: string, level: C
pgsql cluster name, REQUIRED identity parameter
The cluster name will be used as the namespace for PGSQL related resources within that cluster.
The naming needs to follow the specific naming pattern: [a-z][a-z0-9-]* to be compatible with the requirements of different constraints on the identity.
A serial number to identify these instances, unique within its cluster, starting from 0 or 1.
pg_role
name: pg_role, type: enum, level: I
pgsql role, REQUIRED, could be primary,replica,offline
Roles for PGSQL instance, can be: primary, replica, standby or offline.
primary: Primary, there is one and only one primary in a cluster.
replica: Replica for carrying online read-only traffic, there may be a slight replication delay through (10ms~100ms, 100KB).
standby: Special replica that is always synced with primary, there’s no replication delay & data loss on this replica. (currently same as replica)
offline: Offline replica for taking on offline read-only traffic, such as statistical analysis/ETL/personal queries, etc.
Identity params, required params, and instance-level params.
pg_instances
name: pg_instances, type: dict, level: I
define multiple pg instances on node in {port:ins_vars} format.
This parameter is reserved for multi-instance deployment on a single node which is not implemented in Pigsty yet.
pg_upstream
name: pg_upstream, type: ip, level: I
Upstream ip address for standby cluster or cascade replica
Setting pg_upstream is set on primary instance indicate that this cluster is a Standby Cluster, and will receiving changes from upstream instance, thus the primary is actually a standby leader.
Setting pg_upstream for a non-primary instance will explicitly set a replication upstream instance, if it is different from the primary IP Address,
this instance will become a cascade replica. And it’s user’s responsibility to ensure that the upstream IP addr is another instance in the same cluster.
pg_shard
name: pg_shard, type: string, level: C
pgsql shard name, required identity parameter for sharding clusters (e.g. citus cluster), optional for common pgsql clusters.
When multiple pgsql clusters serve the same business together in a horizontally sharding style, Pigsty will mark this group of clusters as a Sharding Group.
pg_shard is the name of the shard group name. It’s usually the prefix of pg_cluster.
For example, if we have a sharding group pg-citus, and 4 clusters in it, there identity params will be:
pgsql shard index number, required identity for sharding clusters, optional for common pgsql clusters.
Sharding cluster index of a sharding group, used in pairs with pg_shard. You can use any non-negative integer as the index number.
gp_role
name: gp_role, type: enum, level: C
greenplum/matrixdb role of this cluster, could be master or segment
master: mark the postgres cluster as greenplum master, which is the default value
segment mark the postgres cluster as greenplum segment
This parameter is only used for greenplum & derived databases, and is ignored for common pgsql cluster.
pg_exporters
name: pg_exporters, type: dict, level: C
additional pg_exporters to monitor remote postgres instances, default values: {}
If you wish to monitor remote postgres instances, define them in pg_exporters and load them with pgsql-monitor.yml playbook.
pg_exporters:# list all remote instances here, alloc a unique unused local port as k20001:{pg_cluster:pg-foo, pg_seq:1, pg_host:10.10.10.10}20004:{pg_cluster:pg-foo, pg_seq:2, pg_host:10.10.10.11}20002:{pg_cluster:pg-bar, pg_seq:1, pg_host:10.10.10.12}20003:{pg_cluster:pg-bar, pg_seq:1, pg_host:10.10.10.13}
set to true to enable offline queries on this instance
default value is false
When this parameter is enabled for a PostgreSQL instance, users belonging to the dbrole_offline group can directly connect to that PostgreSQL instance to perform offline queries (slow queries, interactive queries, ETL/analytical queries).
Instances with this flag are functionally similar to setting pg_role = offline, with the only difference being that offline instances by default do not handle replica service requests, as they exist specifically as dedicated offline/analytical replica instances.
If you don’t have spare instances that can be dedicated to this purpose, you can select a regular replica and enable this parameter at the instance level to accommodate offline queries when needed.
PG_BUSINESS
Database credentials, In-Database Objects that need to be taken care of by Users.
WARNING: YOU HAVE TO CHANGE THESE DEFAULT PASSWORDs in production environment.
# postgres business object definition, overwrite in group varspg_users:[]# postgres business userspg_databases:[]# postgres business databasespg_services:[]# postgres business servicespg_hba_rules:[]# business hba rules for postgrespgb_hba_rules:[]# business hba rules for pgbouncer# global credentials, overwrite in global varspg_dbsu_password:''# dbsu password, empty string means no dbsu password by defaultpg_replication_username:replicatorpg_replication_password:DBUser.Replicatorpg_admin_username:dbuser_dbapg_admin_password:DBUser.DBApg_monitor_username:dbuser_monitorpg_monitor_password:DBUser.Monitor
pg_users
name: pg_users, type: user[], level: C
postgres business users, defined at cluster level.
default values: [], each object in the array defines a User/Role. Examples:
- name:dbuser_meta # REQUIRED, `name` is the only mandatory field of a user definitionpassword:DBUser.Meta # optional, the password, can be a scram-sha-256 hash string or plain textlogin:true# optional, can log in, true by default (new biz ROLE should be false)superuser:false# optional, is superuser? false by defaultcreatedb:false# optional, can create database? false by defaultcreaterole:false# optional, can create role? false by defaultinherit:true# optional, can this role use inherited privileges? true by defaultreplication:false# optional, can this role do replication? false by defaultbypassrls:false# optional, can this role bypass row level security? false by defaultpgbouncer:true# optional, add this user to pgbouncer userlist? false by default (production user should be true explicitly)connlimit:-1# optional, user connection limit, default -1 disable limitexpire_in:3650# optional, now + n days when this role is expired (OVERWRITE expire_at)expire_at:'2030-12-31'# optional, YYYY-MM-DD 'timestamp' when this role is expired (OVERWRITTEN by expire_in)comment:pigsty admin user # optional, comment string for this user/roleroles:[dbrole_admin] # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}parameters:{}# optional, role level parameters with `ALTER ROLE SET`pool_mode:transaction # optional, pgbouncer pool mode at user level, transaction by defaultpool_connlimit:-1# optional, max database connections at user level, default -1 disable limitsearch_path:public # key value config parameters according to postgresql documentation (e.g: use pigsty as default search_path)
The only mandatory field of a user definition is name, and the rest are optional.
pg_databases
name: pg_databases, type: database[], level: C
postgres business databases, defined at cluster level.
default values: [], each object in the array defines a Database. Examples:
- name:meta # REQUIRED, `name` is the only mandatory field of a database definitionbaseline:cmdb.sql # optional, database sql baseline path, (relative path among ansible search path, e.g files/)pgbouncer:true# optional, add this database to pgbouncer database list? true by defaultschemas:[pigsty] # optional, additional schemas to be created, array of schema namesextensions:# optional, additional extensions to be installed: array of `{name[,schema]}`- {name:postgis , schema:public } # You can specify which schema to install the extension in, or leave it unspecified (if unspecified, it will be installed in the first schema of search_path)- {name:timescaledb } # For example, some extensions will create and use fixed schemas, so you don't need to specify a schema.- vector # You can also directly use a string to specify the extension namecomment:pigsty meta database # optional, comment string for this databaseowner:postgres # optional, database owner, postgres by defaulttemplate:template1 # optional, which template to use, template1 by defaultencoding:UTF8 # optional, database encoding, UTF8 by default. (MUST same as template database)locale:C # optional, database locale, C by default. (MUST same as template database)lc_collate:C # optional, database collate, C by default. (MUST same as template database)lc_ctype:C # optional, database ctype, C by default. (MUST same as template database)tablespace:pg_default # optional, default tablespace, 'pg_default' by default.allowconn:true# optional, allow connection, true by default. false will disable connect at allrevokeconn:false# optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)register_datasource:true# optional, register this database to grafana datasources? true by defaultconnlimit:-1# optional, database connection limit, default -1 disable limitpool_auth_user:dbuser_meta # optional, all connection to this pgbouncer database will be authenticated by this userpool_mode:transaction # optional, pgbouncer pool mode at database level, default transactionpool_size:64# optional, pgbouncer pool size at database level, default 64pool_size_reserve:32# optional, pgbouncer pool size reserve at database level, default 32pool_size_min:0# optional, pgbouncer pool size min at database level, default 0pool_max_db_conn:100# optional, max database connections at database level, default 100
In each database definition, the DB name is mandatory and the rest are optional.
pg_services
name: pg_services, type: service[], level: C
postgres business services exposed via haproxy, has to be defined at cluster level.
default values: [], each object in the array defines a Service. Examples:
- name:standby # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standbyport:5435# required, service exposed port (work as kubernetes service node port mode)ip:"*"# optional, service bind ip address, `*` for all ip by defaultselector:"[]"# required, service member selector, use JMESPath to filter inventorydest:default # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by defaultcheck:/sync # optional, health check url path, / by defaultbackup:"[? pg_role == `primary`]"# backup server selectormaxconn:3000# optional, max allowed front-end connectionbalance:roundrobin # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)options:'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'
pg_hba_rules
name: pg_hba_rules, type: hba[], level: C
business hba rules for postgres
default values: [], each object in array is an HBA Rule definition:
Which are array of hba object, each hba object may look like
# RAW HBA RULES- title:allow intranet password accessrole:commonrules:- host all all 10.0.0.0/8 md5- host all all 172.16.0.0/12 md5- host all all 192.168.0.0/16 md5
title: Rule Title, transform into comment in the hba file
rules: Array of strings, each string is a raw hba rule record
role : Applied roles, where to install these hba rules
common: apply for all instances
primary, replica,standby, offline: apply on corresponding instances with that pg_role.
special case: HBA rule with role == 'offline' will be installed on instance with pg_offline_query flag
or you can use another alias form
- addr:'intra'# world|intra|infra|admin|local|localhost|cluster|<cidr>auth:'pwd'# trust|pwd|ssl|cert|deny|<official auth method>user:'all'# all|${dbsu}|${repl}|${admin}|${monitor}|<user>|<group>db:'all'# all|replication|....rules:[]# raw hba string precedence over above alltitle:allow intranet password access
pg_default_hba_rules is similar to this, but is used for global HBA rule settings
pgb_hba_rules
name: pgb_hba_rules, type: hba[], level: C
business hba rules for pgbouncer, default values: []
Similar to pg_hba_rules, array of hba rule object, except this is for pgbouncer.
pg_replication_username
name: pg_replication_username, type: username, level: G
postgres replication username, replicator by default
This parameter is globally used, it is not wise to change it.
pg_replication_password
name: pg_replication_password, type: password, level: G
postgres replication password, DBUser.Replicator by default
WARNING: CHANGE THIS IN PRODUCTION ENVIRONMENT!!!!
pg_admin_username
name: pg_admin_username, type: username, level: G
postgres admin username, dbuser_dba by default, which is a global postgres superuser.
default values: dbuser_dba
pg_admin_password
name: pg_admin_password, type: password, level: G
postgres admin password in plain text, DBUser.DBA by default
WARNING: CHANGE THIS IN PRODUCTION ENVIRONMENT!!!!
pg_monitor_username
name: pg_monitor_username, type: username, level: G
postgres monitor username, dbuser_monitor by default, which is a global monitoring user.
pg_monitor_password
name: pg_monitor_password, type: password, level: G
postgres monitor password, DBUser.Monitor by default.
Try not using the @:/ character in the password to avoid problems with PGURL string.
WARNING: CHANGE THIS IN PRODUCTION ENVIRONMENT!!!!
PostgreSQL dbsu password for pg_dbsu, empty string means no dbsu password, which is the default behavior.
Set this password will allow a well-know dbsu login from remote!
It’s not recommended to set the well-known dbsu (postgres) password for common PGSQL clusters, except for a good reason, such as using pg_mode = citus.
PG_INSTALL
This section is responsible for installing PostgreSQL & Extensions.
If you wish to install a different major version, make sure repo packages exists and overwrite pg_version on cluster level.
To install extra extensions, overwrite pg_extensions on cluster level. Beware that not all extensions are available with other major versions.
pg_dbsu:postgres # os dbsu name, postgres by default, better not change itpg_dbsu_uid:26# os dbsu uid and gid, 26 for default postgres users and groupspg_dbsu_sudo:limit # dbsu sudo privilege, none,limit,all,nopass. limit by defaultpg_dbsu_home:/var/lib/pgsql # postgresql home directory, `/var/lib/pgsql` by defaultpg_dbsu_ssh_exchange:true# exchange postgres dbsu ssh key among same pgsql clusterpg_version:18# postgres major version to be installed, 18 by defaultpg_bin_dir:/usr/pgsql/bin # postgres binary dir, `/usr/pgsql/bin` by defaultpg_log_dir:/pg/log/postgres # postgres log dir, `/pg/log/postgres` by defaultpg_packages:# pg packages to be installed, alias can be used- pgsql-main pgsql-commonpg_extensions:[]# pg extensions to be installed, alias can be used
pg_dbsu
name: pg_dbsu, type: username, level: C
OS DBSU name, postgres by default, it’s not wise to change it.
When installing Greenplum / MatrixDB, set this parameter to the corresponding default value: gpadmin|mxadmin.
pg_dbsu_uid
name: pg_dbsu_uid, type: int, level: C
OS DBSU uid and gid, 26 for default postgres users and groups, which is consistent with the official pgdg RPM.
For Ubuntu/Debian, there’s no default postgres UID/GID, consider using another ad hoc value, such as 543 instead.
pg_dbsu_sudo
name: pg_dbsu_sudo, type: enum, level: C
OS DBSU sudo privilege, could be none, limit ,all ,nopass. limit by default
none: No Sudo privilege
limit: Limited sudo privilege to execute systemctl commands for database-related components, default.
all: Full sudo privilege, password required.
nopass: Full sudo privileges without a password (not recommended).
default values: limit, which only allow sudo systemctl <start|stop|reload> <postgres|patroni|pgbouncer|...>
Available sudo services:
patroni
pgbouncer
postgres
pg_exporter
pgbackrest
pgbouncer_exporter
pgbackrest_exporter
vip-manager
haproxy (reload only)
pg_dbsu_home
name: pg_dbsu_home, type: path, level: C
postgresql home directory, /var/lib/pgsql by default, which is consistent with the official pgdg RPM.
pg_dbsu_ssh_exchange
name: pg_dbsu_ssh_exchange, type: bool, level: C
exchange postgres os dbsu ssh key among pgsql instances?
default value is true, means the dbsu can ssh to each other among the playbook execution hosts.
For scenarios where ssh access is strictly limited, you can set it to false.
Please note that SSH key exchange occurs between instances that are executing the same playbook. If you run the pgsql role for a single PostgreSQL cluster, the key exchange will occur between all instances in that cluster.
If you run the pgsql role for all PostgreSQL clusters, the key exchange will occur between all instances, which can lead to severe combinatorial explosions for large clusters.
If any instance involved in the key exchange does not have the pg_dbsu user, the key exchange will fail for that instance, but will not affect other instances.
pg_version
name: pg_version, type: enum, level: C
postgres major version to be installed, 18 by default
Note that PostgreSQL physical stream replication cannot cross major versions, so do not configure this on instance level.
You can use the parameters in pg_packages and pg_extensions to install rpm/deb for the specific pg major version.
pg_bin_dir
name: pg_bin_dir, type: path, level: C
postgres binary dir, /usr/pgsql/bin by default
The default value is a soft link created manually during the installation process, pointing to the specific Postgres version dir installed.
For example /usr/pgsql -> /usr/pgsql-17. For more details, check PGSQL File Structure for details.
pg_log_dir
name: pg_log_dir, type: path, level: C
postgres log dir, /pg/log/postgres by default.
caveat: if pg_log_dir is prefixed with pg_data it will not be created explicitly (it will be created by postgres itself then).
pg_packages
name: pg_packages, type: string[], level: C
PostgreSQL packages (rpm/deb) to be installed. This is an array of package names, where each element is a comma or space-separated list of PG package names or aliases.
Default value: [ pgsql-main pgsql-common ]
These default values are two aliases that are translated through alias mapping into the main RPM/DEB package names for the current PG major version, as well as version-independent common components (such as Patroni, PgBackrest, etc.)
Since Pigsty v3, you can use the alias lists specified in the system configuration in roles/node_id/vars for this parameter.
The advantage of using package aliases is that you don’t need to worry about package names, architectures, and major version numbers for PostgreSQL-related packages across different system platforms, thus abstracting away differences between operating systems:
Packages defined here will first be translated through the package_map, then undergo PG major version number substitution, and finally install the actual RPM/DEB packages.
You can also directly specify the final RPM/DEB package names to be installed, where version placeholders like ${pg_version} or $v in the package name will be replaced with the specific major version number pg_version.
pg_extensions
name: pg_extensions, type: string[], level: C
PG extensions to be installed (rpm/deb), this is an array of software package names, each element is a comma or space separated PG extension package name.
This parameter is similar to pg_packages, but is usually used to specify the extension to be installed @ global | cluster level, and the software packages specified here will be upgraded to the latest available version.
The default value of this parameter is the three most important extension plugins in the PG extension ecosystem: postgis, timescaledb, pgvector.
pg_extensions:[]
The complete list of extensions can be found in auto generated config
It also init cluster template databases with default roles, schemas & extensions & default privileges specified in PG_PROVISION
pg_data:/pg/data # postgres data directory, `/pg/data` by defaultpg_fs_main:/data/postgres # postgres main data directory, `/data/postgres` by defaultpg_fs_backup:/data/backups # postgres backup data directory, `/data/backups` by defaultpg_storage_type:SSD # storage type for pg main data, SSD,HDD, SSD by defaultpg_dummy_filesize:64MiB # size of `/pg/dummy`, hold 64MB disk space for emergency usepg_listen:'0.0.0.0'# postgres/pgbouncer listen addresses, comma separated listpg_port:5432# postgres listen port, 5432 by defaultpg_localhost:/var/run/postgresql# postgres unix socket dir for localhost connectionpatroni_enabled:true# if disabled, no postgres cluster will be created during initpatroni_mode:default # patroni working mode: default,pause,removepg_namespace:/pg # top level key namespace in etcd, used by patroni & vippatroni_port:8008# patroni listen port, 8008 by defaultpatroni_log_dir:/pg/log/patroni # patroni log dir, `/pg/log/patroni` by defaultpatroni_ssl_enabled:false# secure patroni RestAPI communications with SSL?patroni_watchdog_mode:off# patroni watchdog mode: automatic, required, off. off by defaultpatroni_username:postgres # patroni restapi username, `postgres` by defaultpatroni_password:Patroni.API # patroni restapi password, `Patroni.API` by defaultpg_primary_db:postgres # primary database name, used by citus,etc... postgres by defaultpg_parameters:{}# extra parameters in postgresql.auto.confpg_files:[]# extra files to be copied to postgres data directory (e.g. license)pg_conf:oltp.yml # config template: oltp,olap,crit,tiny. `oltp.yml` by defaultpg_max_conn:auto # postgres max connections, `auto` will use recommended valuepg_shared_buffer_ratio:0.25# postgres shared buffers ratio, 0.25 by default, 0.1~0.4pg_rto:30# recovery time objective in seconds, `30s` by defaultpg_rpo:1048576# recovery point objective in bytes, `1MiB` at most by defaultpg_libs:'pg_stat_statements, auto_explain'# preloaded libraries, `pg_stat_statements,auto_explain` by defaultpg_delay:0# replications apply delay for standby cluster leaderpg_checksum:true# enable data checksum for postgres cluster?pg_pwd_enc:scram-sha-256 # passwords encryption algorithm: md5,scram-sha-256pg_encoding:UTF8 # database cluster encoding, `UTF8` by defaultpg_locale:C # database cluster local, `C` by defaultpg_lc_collate:C # database cluster collate, `C` by defaultpg_lc_ctype:C # database character type, `C` by default#pgsodium_key: "" # pgsodium key, 64 hex digits, default to sha256(pg_cluster)#pgsodium_getkey_script: "" # pgsodium getkey script path, pgsodium_getkey by default
pg_data
name: pg_data, type: path, level: C
postgres data directory, /pg/data by default
default values: /pg/data, DO NOT CHANGE IT.
It’s a soft link that points to the underlying data directory.
pg_fs_main
name: pg_fs_main, type: path, level: C
postgres main data directory, /data/postgres by default.
This directory will be created and owned by the pg_dbsu user, and it will be used as the main data directory for postgres.
If your main data directory node_data is changed, consider changing this parameter as well.
It’s recommended to use NVME SSD for postgres main data storage, Pigsty is optimized for SSD storage by default.
If you are using HDD storage, consider changing the pg_storage_type to HDD to optimize for HDD storage.
pg_fs_backup
name: pg_fs_backup, type: path, level: C
postgres backup data directory, /data/backups by default
This directory will be created and owned by the pg_dbsu user, and it will be used as the local backup storage for postgres.
Local backup is enabled by default on primary pg cluster.
If you are using the default pgbackrest_method = local, it is recommended to have a separate disk for backup storage.
The backup disk should be large enough to hold all your backups, at least enough for 3 base backups + 2-day WAL archive.
This is usually not a problem since you can use affordable and large HDD for that.
It’s optional if you are using remote / centralized backup storage (e.g. pgbackrest_method = minio).
pg_storage_type
name: pg_storage_type, type: enum, level: C
storage type for pg main data, SSD,HDD, SSD by default
default values: SSD, it will affect some tuning parameters, such as random_page_cost & effective_io_concurrency
pg_dummy_filesize
name: pg_dummy_filesize, type: size, level: C
size of /pg/dummy, default values: 64MiB, which hold 64MB disk space for emergency use
When the disk is full, removing the placeholder file can free up some space for emergency use, it is recommended to set at least 8GiB for production use.
pg_listen
name: pg_listen, type: ip, level: C
postgres/pgbouncer listen address, 0.0.0.0 (all ipv4 addr) by default
You can use placeholder in this variable:
${ip}: translate to inventory_hostname, which is primary private IP address in the inventory
This parameter is used to specify and manage configuration parameters in postgresql.auto.conf.
After all instances in the cluster have completed initialization, the pg_param task will sequentially overwrite the key/value pairs in this dictionary to /pg/data/postgresql.auto.conf.
Note: Please do not manually modify this configuration file, or use ALTER SYSTEM to change cluster configuration parameters. Any changes will be overwritten during the next configuration sync.
This variable has a higher priority than the cluster configuration in Patroni/DCS (i.e., it has a higher priority than the cluster configuration edited by Patroni edit-config). Therefore, it can typically override the cluster default parameters at the instance level.
When your cluster members have different specifications (not recommended!), you can fine-tune the configuration of each instance using this parameter.
Please note that some important cluster parameters (which have requirements for primary and replica parameter values) are managed directly by Patroni through command-line parameters and have the highest priority.
These cannot be overridden by this method. For these parameters, you must use Patroni edit-config for management and configuration.
PostgreSQL parameters that must remain consistent across primary and replicas (inconsistency will prevent the replica from starting!):
wal_level
max_connections
max_locks_per_transaction
max_worker_processes
max_prepared_transactions
track_commit_timestamp
Parameters that should ideally remain consistent across primary and replicas (considering the possibility of primary-replica switch):
listen_addresses
port
cluster_name
hot_standby
wal_log_hints
max_wal_senders
max_replication_slots
wal_keep_segments
wal_keep_size
You can set non-existent parameters (such as GUCs from extensions), but changing existing configurations to illegal values may prevent PostgreSQL from starting. Please configure with caution!
pg_files
Parameter Name: pg_files, Type: path[], Level: C
Designates a list of files to be copied to the {{ pg_data }} directory. The default value is an empty array: [].
Files specified in this parameter will be copied to the {{ pg_data }} directory. This is mainly used to distribute license files required by special commercial versions of the PostgreSQL kernel.
Currently, only the PolarDB (Oracle-compatible) kernel requires a license file. For example, you can place the license.lic file in the files/ directory and specify it in pg_files:
pg_files:[license.lic ]
pg_conf
name: pg_conf, type: enum, level: C
config template: {oltp,olap,crit,tiny}.yml, oltp.yml by default
tiny.yml: optimize for tiny nodes, virtual machines, small demo, (18Core, 116GB)
oltp.yml: optimize for OLTP workloads and latency-sensitive applications, (4C8GB+), which is the default template
olap.yml: optimize for OLAP workloads and throughput (4C8G+)
crit.yml: optimize for data consistency and critical applications (4C8G+)
default values: oltp.yml, but configure procedure will set this value to tiny.yml if current node is a tiny node.
You can have your own template, just put it under templates/<mode>.yml and set this value to the template name.
pg_max_conn
name: pg_max_conn, type: int, level: C
postgres max connections, You can specify a value between 50 and 5000, or use auto to use recommended value.
It’s not recommended to set this value greater than 5000, otherwise you have to increase the haproxy service connection limit manually as well.
Pgbouncer’s transaction pooling can alleviate the problem of too many OLTP connections, but it’s not recommended to use it in OLAP scenarios.
pg_shared_buffer_ratio
name: pg_shared_buffer_ratio, type: float, level: C
postgres shared buffer memory ratio, 0.25 by default, 0.1~0.4
default values: 0.25, means 25% of node memory will be used as PostgreSQL shard buffers.
Setting this value greater than 0.4 (40%) is usually not a good idea.
Note that shared buffer is only part of shared memory in PostgreSQL, to calculate the total shared memory, use show shared_memory_size_in_huge_pages;.
pg_rto
name: pg_rto, type: int, level: C
recovery time objective in seconds, This will be used as Patroni TTL value, 30s by default.
If a primary instance is missing for such a long time, a new leader election will be triggered.
Decreasing the value can reduce the unavailable time (unable to write) of the cluster during failover, but it will make the cluster more sensitive to network jitter, thus increase the chance of false-positive failover.
Config this according to your network condition and expectation to trade-off between chance and impact, the default value is 30s, and it will be populated to the following patroni parameters:
# the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30ttl:{{pg_rto }}# the number of seconds the loop will sleep. Default value: 10 , this is patroni check loop intervalloop_wait:{{(pg_rto / 3)|round(0, 'ceil')|int }}# timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10retry_timeout:{{(pg_rto / 3)|round(0, 'ceil')|int }}# the amount of time a primary is allowed to recover from failures before failover is triggered (in seconds), Max RTO: 2 loop wait + primary_start_timeoutprimary_start_timeout:{{(pg_rto / 3)|round(0, 'ceil')|int }}
pg_rpo
name: pg_rpo, type: int, level: C
recovery point objective in bytes, 1MiB at most by default
default values: 1048576, which will tolerate at most 1MiB data loss during failover.
when the primary is down and all replicas are lagged, you have to make a tough choice to trade off between Availability and Consistency:
Promote a replica to be the new primary and bring the system back online ASAP, with the price of an acceptable data loss (e.g. less than 1MB).
Wait for the primary to come back (which may never be) or human intervention to avoid any data loss.
You can use crit.ymlconf template to ensure no data loss during failover, but it will sacrifice some performance.
pg_libs
name: pg_libs, type: string, level: C
shared preloaded libraries, pg_stat_statements,auto_explain by default.
They are two extensions that come with PostgreSQL, and it is strongly recommended to enable them.
For existing clusters, you can configure the shared_preload_libraries parameter of the cluster and apply it.
If you want to use TimescaleDB or Citus extensions, you need to add timescaledb or citus to this list. timescaledb and citus should be placed at the top of this list, for example:
citus,timescaledb,pg_stat_statements,auto_explain
Other extensions that need to be loaded can also be added to this list, such as pg_cron, pgml, etc.
Generally, citus and timescaledb have the highest priority and should be added to the top of the list.
pg_delay
name: pg_delay, type: interval, level: I
replications apply delay for standby cluster leader, default values: 0.
if this value is set to a positive value, the standby cluster leader will be delayed for this time before apply WAL changes.
enable data checksum for postgres cluster? The v3.7.0 default value is true.
This parameter can only be set before PGSQL deployment. (but you can enable it manually later)
If pg_confcrit.yml template is used, data checksum is always enabled regardless of this parameter to ensure data integrity.
pg_pwd_enc
name: pg_pwd_enc, type: enum, level: C
password encryption algorithm: md5, scram-sha-256
default values: scram-sha-256, if you have compatibility issues with old clients, you can set it to md5 instead.
md5 encryption is deprecated!
The md5 option is deprecated but remains available in v3.7.0 for legacy clients; prefer scram-sha-256.
pg_encoding
name: pg_encoding, type: enum, level: C
database cluster encoding, UTF8 by default
pg_locale
name: pg_locale, type: enum, level: C
The locale set for PostgreSQL, default is C.
When configure detects that the current PG version is greater than or equal to 17, or the current system explicitly supports C.utf8, it will automatically configure this parameter to C.UTF-8.
When the PostgreSQL version is greater than or equal to 17, the C and C.UTF-8 configurations will use the PostgreSQL internal Locale Provider.
Unless you are very clear about what you are doing, it is strongly recommended to use the default C or C.UTF-8 configuration.
pg_lc_collate
name: pg_lc_collate, type: enum, level: C
The locale set for PostgreSQL, default is C.
When configure detects that the current PG version is greater than or equal to 17, or the current system explicitly supports C.utf8, it will automatically configure this parameter to C.UTF-8.
Unless you are very clear about what you are doing, it is strongly recommended to use the default C or C.UTF-8 configuration.
The parameter behaves like pg_locale, but for collate.
pg_lc_ctype
name: pg_lc_ctype, type: enum, level: C
The locale set for PostgreSQL, default is C.
When configure detects that the current PG version is greater than or equal to 17, or the current system explicitly supports C.utf8, it will automatically configure this parameter to C.UTF-8.
When the PostgreSQL version is greater than or equal to 17, the C and C.UTF-8 configurations will use the PostgreSQL internal Locale Provider.
This parameter behaves like pg_locale, but for ctype.
Unless you are very clear about what you are doing, it is strongly recommended to use the default C or C.UTF-8 configuration.
pgsodium_key
name: pgsodium_key, type: string, level: C
Default value is not defined, which will use the SHA256 hash of the pg_cluster as the key.
You can provide a custom pgsodium key, which should be a 64 hex digit string.
The key will be written to /pg/conf/pgsodium.key.
pgsodium_getkey_script
name: pgsodium_getkey_script, type: path, level: C
The default getkey script will just read the pgsodium_key from /pg/conf/pgsodium.key, and return it.
If your key is managed by external system like KMS, IAM, …, you can implement your own getkey script to fetch the key from there: examples.
PG_PROVISION
PG_BOOTSTRAP will bootstrap a new postgres cluster with patroni, while PG_PROVISION will create default objects in the cluster, including:
pg_provision:true# provision postgres cluster after bootstrappg_init:pg-init # provision init script for cluster template, `pg-init` by defaultpg_default_roles:# default roles and users in postgres cluster- {name:dbrole_readonly ,login:false ,comment:role for global read-only access }- {name:dbrole_offline ,login:false ,comment:role for restricted read-only access }- {name:dbrole_readwrite ,login:false ,roles:[dbrole_readonly] ,comment:role for global read-write access }- {name:dbrole_admin ,login:false ,roles:[pg_monitor, dbrole_readwrite] ,comment:role for object creation }- {name:postgres ,superuser:true ,comment:system superuser }- {name:replicator ,replication:true ,roles:[pg_monitor, dbrole_readonly] ,comment:system replicator }- {name:dbuser_dba ,superuser:true ,roles:[dbrole_admin] ,pgbouncer:true ,pool_mode:session, pool_connlimit:16 , comment:pgsql admin user }- {name:dbuser_monitor ,roles:[pg_monitor, dbrole_readonly] ,pgbouncer:true ,parameters:{log_min_duration_statement:1000 } ,pool_mode:session ,pool_connlimit:8 ,comment:pgsql monitor user }pg_default_privileges:# default privileges when created by admin user- GRANT USAGE ON SCHEMAS TO dbrole_readonly- GRANT SELECT ON TABLES TO dbrole_readonly- GRANT SELECT ON SEQUENCES TO dbrole_readonly- GRANT EXECUTE ON FUNCTIONS TO dbrole_readonly- GRANT USAGE ON SCHEMAS TO dbrole_offline- GRANT SELECT ON TABLES TO dbrole_offline- GRANT SELECT ON SEQUENCES TO dbrole_offline- GRANT EXECUTE ON FUNCTIONS TO dbrole_offline- GRANT INSERT ON TABLES TO dbrole_readwrite- GRANT UPDATE ON TABLES TO dbrole_readwrite- GRANT DELETE ON TABLES TO dbrole_readwrite- GRANT USAGE ON SEQUENCES TO dbrole_readwrite- GRANT UPDATE ON SEQUENCES TO dbrole_readwrite- GRANT TRUNCATE ON TABLES TO dbrole_admin- GRANT REFERENCES ON TABLES TO dbrole_admin- GRANT TRIGGER ON TABLES TO dbrole_admin- GRANT CREATE ON SCHEMAS TO dbrole_adminpg_default_schemas:[monitor ] # default schemas to be createdpg_default_extensions:# default extensions to be created- {name:pg_stat_statements ,schema:monitor }- {name:pgstattuple ,schema:monitor }- {name:pg_buffercache ,schema:monitor }- {name:pageinspect ,schema:monitor }- {name:pg_prewarm ,schema:monitor }- {name:pg_visibility ,schema:monitor }- {name:pg_freespacemap ,schema:monitor }- {name:postgres_fdw ,schema:public }- {name:file_fdw ,schema:public }- {name:btree_gist ,schema:public }- {name:btree_gin ,schema:public }- {name:pg_trgm ,schema:public }- {name:intagg ,schema:public }- {name:intarray ,schema:public }- {name:pg_repack }pg_reload:true# reload postgres/pgbouncer/vip after conf changespg_default_hba_rules:# postgres default host-based authentication rules- {user:'${dbsu}',db:all ,addr:local ,auth:ident ,title:'dbsu access via local os user ident'}- {user:'${dbsu}',db:replication ,addr:local ,auth:ident ,title:'dbsu replication from local os ident'}- {user:'${repl}',db:replication ,addr:localhost ,auth:pwd ,title:'replicator replication from localhost'}- {user:'${repl}',db:replication ,addr:intra ,auth:pwd ,title:'replicator replication from intranet'}- {user:'${repl}',db:postgres ,addr:intra ,auth:pwd ,title:'replicator postgres db from intranet'}- {user:'${monitor}',db:all ,addr:localhost ,auth:pwd ,title:'monitor from localhost with password'}- {user:'${monitor}',db:all ,addr:infra ,auth:pwd ,title:'monitor from infra host with password'}- {user:'${admin}',db:all ,addr:infra ,auth:ssl ,title:'admin @ infra nodes with pwd & ssl'}- {user:'${admin}',db:all ,addr:world ,auth:ssl ,title:'admin @ everywhere with ssl & pwd'}- {user:'+dbrole_readonly',db:all ,addr:localhost ,auth:pwd ,title:'pgbouncer read/write via local socket'}- {user:'+dbrole_readonly',db:all ,addr:intra ,auth:pwd ,title:'read/write biz user via password'}- {user:'+dbrole_offline' ,db:all ,addr:intra ,auth:pwd ,title:'allow etl offline tasks from intranet'}pgb_default_hba_rules:# pgbouncer default host-based authentication rules- {user:'${dbsu}',db:pgbouncer ,addr:local ,auth:peer ,title:'dbsu local admin access with os ident'}- {user:'all' ,db:all ,addr:localhost ,auth:pwd ,title:'allow all user local access with pwd'}- {user:'${monitor}',db:pgbouncer ,addr:intra ,auth:pwd ,title:'monitor access via intranet with pwd'}- {user:'${monitor}',db:all ,addr:world ,auth:deny ,title:'reject all other monitor access addr'}- {user:'${admin}',db:all ,addr:intra ,auth:pwd ,title:'admin access via intranet with pwd'}- {user:'${admin}',db:all ,addr:world ,auth:deny ,title:'reject all other admin access addr'}- {user:'all' ,db:all ,addr:intra ,auth:pwd ,title:'allow all user intra access with pwd'}
pg_provision
name: pg_provision, type: bool, level: C
provision postgres cluster after bootstrap, default value is true.
If disabled, postgres cluster will not be provisioned after bootstrap.
pg_default_privileges:# default privileges when created by admin user- GRANT USAGE ON SCHEMAS TO dbrole_readonly- GRANT SELECT ON TABLES TO dbrole_readonly- GRANT SELECT ON SEQUENCES TO dbrole_readonly- GRANT EXECUTE ON FUNCTIONS TO dbrole_readonly- GRANT USAGE ON SCHEMAS TO dbrole_offline- GRANT SELECT ON TABLES TO dbrole_offline- GRANT SELECT ON SEQUENCES TO dbrole_offline- GRANT EXECUTE ON FUNCTIONS TO dbrole_offline- GRANT INSERT ON TABLES TO dbrole_readwrite- GRANT UPDATE ON TABLES TO dbrole_readwrite- GRANT DELETE ON TABLES TO dbrole_readwrite- GRANT USAGE ON SEQUENCES TO dbrole_readwrite- GRANT UPDATE ON SEQUENCES TO dbrole_readwrite- GRANT TRUNCATE ON TABLES TO dbrole_admin- GRANT REFERENCES ON TABLES TO dbrole_admin- GRANT TRIGGER ON TABLES TO dbrole_admin- GRANT CREATE ON SCHEMAS TO dbrole_admin
Pigsty has a built-in privileges based on the default role system, check PGSQL Privileges for details.
postgres default host-based authentication rules, array of hba rule object.
default value provides a fair enough security level for common scenarios, check PGSQL Authentication for details.
pg_default_hba_rules:# postgres default host-based authentication rules- {user:'${dbsu}',db:all ,addr:local ,auth:ident ,title:'dbsu access via local os user ident'}- {user:'${dbsu}',db:replication ,addr:local ,auth:ident ,title:'dbsu replication from local os ident'}- {user:'${repl}',db:replication ,addr:localhost ,auth:pwd ,title:'replicator replication from localhost'}- {user:'${repl}',db:replication ,addr:intra ,auth:pwd ,title:'replicator replication from intranet'}- {user:'${repl}',db:postgres ,addr:intra ,auth:pwd ,title:'replicator postgres db from intranet'}- {user:'${monitor}',db:all ,addr:localhost ,auth:pwd ,title:'monitor from localhost with password'}- {user:'${monitor}',db:all ,addr:infra ,auth:pwd ,title:'monitor from infra host with password'}- {user:'${admin}',db:all ,addr:infra ,auth:ssl ,title:'admin @ infra nodes with pwd & ssl'}- {user:'${admin}',db:all ,addr:world ,auth:ssl ,title:'admin @ everywhere with ssl & pwd'}- {user:'+dbrole_readonly',db:all ,addr:localhost ,auth:pwd ,title:'pgbouncer read/write via local socket'}- {user:'+dbrole_readonly',db:all ,addr:intra ,auth:pwd ,title:'read/write biz user via password'}- {user:'+dbrole_offline' ,db:all ,addr:intra ,auth:pwd ,title:'allow etl offline tasks from intranet'}
pgbouncer default host-based authentication rules, array or hba rule object.
default value provides a fair enough security level for common scenarios, check PGSQL Authentication for details.
pgb_default_hba_rules:# pgbouncer default host-based authentication rules- {user:'${dbsu}',db:pgbouncer ,addr:local ,auth:peer ,title:'dbsu local admin access with os ident'}- {user:'all' ,db:all ,addr:localhost ,auth:pwd ,title:'allow all user local access with pwd'}- {user:'${monitor}',db:pgbouncer ,addr:intra ,auth:pwd ,title:'monitor access via intranet with pwd'}- {user:'${monitor}',db:all ,addr:world ,auth:deny ,title:'reject all other monitor access addr'}- {user:'${admin}',db:all ,addr:intra ,auth:pwd ,title:'admin access via intranet with pwd'}- {user:'${admin}',db:all ,addr:world ,auth:deny ,title:'reject all other admin access addr'}- {user:'all' ,db:all ,addr:intra ,auth:pwd ,title:'allow all user intra access with pwd'}
PG_BACKUP
This section defines variables for pgBackRest, which is used for PGSQL PITR (Point-In-Time-Recovery).
pgbackrest_enabled:true# enable pgbackrest on pgsql host?pgbackrest_clean:true# remove pg backup data during init?pgbackrest_log_dir:/pg/log/pgbackrest# pgbackrest log dir, `/pg/log/pgbackrest` by defaultpgbackrest_method:local # pgbackrest repo method: local,minio,[user-defined...]pgbackrest_init_backup:true# take a full backup after pgbackrest is initialized?pgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositorylocal:# default pgbackrest repo with local posix fspath:/pg/backup # local backup directory, `/pg/backup` by defaultretention_full_type:count # retention full backups by countretention_full:2# keep 2, at most 3 full backups when using local fs repominio:# optional minio repo for pgbackresttype:s3 # minio is s3-compatible, so s3 is useds3_endpoint:sss.pigsty # minio endpoint domain name, `sss.pigsty` by defaults3_region:us-east-1 # minio region, us-east-1 by default, useless for minios3_bucket:pgsql # minio bucket name, `pgsql` by defaults3_key:pgbackrest # minio user access key for pgbackrests3_key_secret:S3User.Backup # minio user secret key for pgbackrests3_uri_style:path # use path style uri for minio rather than host stylepath:/pgbackrest # minio backup path, default is `/pgbackrest`storage_port:9000# minio port, 9000 by defaultstorage_ca_file:/etc/pki/ca.crt # minio ca file path, `/etc/pki/ca.crt` by defaultblock:y# Enable block incremental backupbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest # AES encryption password, default is 'pgBackRest'retention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for the last 14 days
pgbackrest_enabled
name: pgbackrest_enabled, type: bool, level: C
enable pgBackRest on pgsql host? default value is true
When using the local file system backup repository (local), only the primary instance of the cluster will actually enable pgbackrest. Other instances will only initialize an empty repository.
pgbackrest_clean
name: pgbackrest_clean, type: bool, level: C
remove pg backup data during init? default value is true
pgbackrest_log_dir
name: pgbackrest_log_dir, type: path, level: C
pgBackRest log dir, /pg/log/pgbackrest by default, which is referenced by promtail the logging agent.
pgbackrest_method
name: pgbackrest_method, type: enum, level: C
pgBackRest repo method: local, minio, or other user-defined methods, local by default
This parameter is used to determine which repo to use for pgBackRest, all available repo methods are defined in pgbackrest_repo.
Pigsty will use local backup repo by default, which will create a backup repo on primary instance’s /pg/backup directory. The underlying storage is specified by pg_fs_backup.
pgbackrest_init_backup
name: pgbackrest_init_backup, type: bool, level: C
Take a full backup after pgBackRest is initialized? default value is true.
An initial pgbackrest backup is created after repo init if:
pgbackrest_init_backup is true (and pgbackrest_enabled is true of course)
The /etc/pgbackrest/initial.done marker file doesn’t exist (will be created after the initial backup is done).
If you don’t want to take an initial full backup at all, just set this parameter tofalse.
default value includes two repo methods: local and minio, which are defined as follows:
pgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositorylocal:# default pgbackrest repo with local posix fspath:/pg/backup # local backup directory, `/pg/backup` by defaultretention_full_type:count # retention full backups by countretention_full:2# keep 2, at most 3 full backups when using local fs repominio:# optional minio repo for pgbackresttype:s3 # minio is s3-compatible, so s3 is useds3_endpoint:sss.pigsty # minio endpoint domain name, `sss.pigsty` by defaults3_region:us-east-1 # minio region, us-east-1 by default, useless for minios3_bucket:pgsql # minio bucket name, `pgsql` by defaults3_key:pgbackrest # minio user access key for pgbackrests3_key_secret:S3User.Backup # minio user secret key for pgbackrests3_uri_style:path # use path style uri for minio rather than host stylepath:/pgbackrest # minio backup path, default is `/pgbackrest`storage_port:9000# minio port, 9000 by defaultstorage_ca_file:/etc/pki/ca.crt # minio ca file path, `/etc/pki/ca.crt` by defaultblock:y# Enable block incremental backupbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest # AES encryption password, default is 'pgBackRest'retention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for the last 14 days
You can define a new backup repository, for example, using AWS S3, GCP or another cloud provider’s S3-compatible storage service.
In the backup repository definition parameters, you can use ${pg_cluster} variable to reference the cluster name, for example, as part of the backup path or encryption key.
But if you have cross-cluster PITR requirements, you should keep the backup repository path and encryption key the same.
PG_ACCESS
This section is about exposing PostgreSQL service to the outside world: including:
Connection Pooling with pgbouncer
Exposing different PostgreSQL services on different ports with haproxy
Bind an optional L2 VIP to the primary instance with vip-manager
Register cluster/instance DNS records with to dnsmasq on infra nodes
pgbouncer_enabled:true# if disabled, pgbouncer will not be launched on pgsql hostpgbouncer_port:6432# pgbouncer listen port, 6432 by defaultpgbouncer_log_dir:/pg/log/pgbouncer # pgbouncer log dir, `/pg/log/pgbouncer` by defaultpgbouncer_auth_query:false# query postgres to retrieve unlisted business users?pgbouncer_poolmode:transaction # pooling mode: transaction,session,statement, transaction by defaultpgbouncer_sslmode:disable # pgbouncer client ssl mode, disable by defaultpg_weight:100#INSTANCE # relative load balance weight in service, 100 by default, 0-255pg_default_service_dest:pgbouncer# default service destination if svc.dest='default'pg_default_services:# postgres default service definitions- {name:primary ,port:5433 ,dest:default ,check:/primary ,selector:"[]"}- {name:replica ,port:5434 ,dest:default ,check:/read-only ,selector:"[]", backup:"[? pg_role == `primary` || pg_role == `offline` ]"}- {name:default ,port:5436 ,dest:postgres ,check:/primary ,selector:"[]"}- {name:offline ,port:5438 ,dest:postgres ,check:/replica ,selector:"[? pg_role == `offline` || pg_offline_query ]", backup:"[? pg_role == `replica` && !pg_offline_query]"}pg_vip_enabled:false# enable a l2 vip for pgsql primary? false by defaultpg_vip_address:127.0.0.1/24 # vip address in `<ipv4>/<mask>` format, require if vip is enabledpg_vip_interface:eth0 # vip network interface to listen, eth0 by defaultpg_dns_suffix:''# pgsql dns suffix, '' by defaultpg_dns_target:auto # auto, primary, vip, none, or ad hoc ip
pgbouncer_enabled
name: pgbouncer_enabled, type: bool, level: C
default value is true, if disabled, pgbouncer will not be launched on pgsql host
pgbouncer_port
name: pgbouncer_port, type: port, level: C
pgbouncer listen port, 6432 by default
pgbouncer_log_dir
name: pgbouncer_log_dir, type: path, level: C
pgbouncer log dir, /pg/log/pgbouncer by default, referenced by promtail the logging agent.
pgbouncer_auth_query
name: pgbouncer_auth_query, type: bool, level: C
query postgres to retrieve unlisted business users? default value is false
If enabled, pgbouncer user will be authenticated against postgres databases with SELECT username, password FROM monitor.pgbouncer_auth($1), otherwise, only the users with pgbouncer: true will be allowed to connect to pgbouncer.
pgbouncer_poolmode
name: pgbouncer_poolmode, type: enum, level: C
Pgbouncer pooling mode: transaction, session, statement, transaction by default
session: Session-level pooling with the best compatibility.
transaction: Transaction-level pooling with better performance (lots of small conns), could break some session level features such as notify/listen, etc…
statements: Statement-level pooling which is used for simple read-only queries.
If your application has some compatibility issues with pgbouncer, you can try to change this value to session instead.
pgbouncer_sslmode
name: pgbouncer_sslmode, type: enum, level: C
pgbouncer client ssl mode, disable by default
default values: disable, beware that this may have a huge performance impact on your pgbouncer.
disable: Plain TCP. If a client requests TLS, it’s ignored. Default.
allow: If a client requests TLS, it is used. If not, plain TCP is used. If the client presents a client certificate, it is not validated.
prefer: Same as allow.
require: Client must use TLS. If not, the client connection is rejected. If the client presents a client certificate, it is not validated.
verify-ca: Client must use TLS with valid client certificate.
default value is false, means no L2 VIP is created for this cluster.
L2 VIP can only be used in the same L2 network, which may incur extra restrictions on your network topology.
pg_vip_address
name: pg_vip_address, type: cidr4, level: C
vip address in <ipv4>/<mask> format, if vip is enabled, this parameter is required.
default values: 127.0.0.1/24. This value is consist of two parts: ipv4 and mask, separated by /.
pg_vip_interface
name: pg_vip_interface, type: string, level: C/I
vip network interface to listen, eth0 by default.
It should be the same primary intranet interface of your node, which is the IP address you used in the inventory file.
If your nodes have different interface, you can override it on instance vars:
pg-test:hosts:10.10.10.11:{pg_seq:1, pg_role:replica ,pg_vip_interface:eth0 }10.10.10.12:{pg_seq:2, pg_role:primary ,pg_vip_interface:eth1 }10.10.10.13:{pg_seq:3, pg_role:replica ,pg_vip_interface:eth2 }vars:pg_vip_enabled:true# enable L2 VIP for this cluster, bind to primary instance by defaultpg_vip_address:10.10.10.3/24# the L2 network CIDR: 10.10.10.0/24, the vip address: 10.10.10.3# pg_vip_interface: eth1 # if your node have non-uniform interface, you can define it here
pg_dns_suffix
name: pg_dns_suffix, type: string, level: C
pgsql dns suffix, empty string by default, cluster DNS name is defined as {{ pg_cluster }}{{ pg_dns_suffix }}
For example, if you set pg_dns_suffix to .db.vip.company.tld for cluster pg-test, then the cluster DNS name will be pg-test.db.vip.company.tld
pg_dns_target
name: pg_dns_target, type: enum, level: C
Could be: auto, primary, vip, none, or an ad hoc ip address, which will be the target IP address of cluster DNS record.
default values: auto , which will bind to pg_vip_address if pg_vip_enabled, or fallback to cluster primary instance ip address.
vip: bind to pg_vip_address
primary: resolve to cluster primary instance ip address
auto: resolve to pg_vip_address if pg_vip_enabled, or fallback to cluster primary instance ip address.
none: do not bind to any ip address
<ipv4>: bind to the given IP address
PG_MONITOR
pg_exporter_enabled:true# enable pg_exporter on pgsql hosts?pg_exporter_config:pg_exporter.yml # pg_exporter configuration file namepg_exporter_cache_ttls:'1,10,60,300'# pg_exporter collector ttl stage in seconds, '1,10,60,300' by defaultpg_exporter_port:9630# pg_exporter listen port, 9630 by defaultpg_exporter_params:'sslmode=disable'# extra url parameters for pg_exporter dsnpg_exporter_url:''# overwrite auto-generate pg dsn if specifiedpg_exporter_auto_discovery:true# enable auto database discovery? enabled by defaultpg_exporter_exclude_database:'template0,template1,postgres'# csv of databases that WILL NOT be monitored during auto-discoverypg_exporter_include_database:''# csv of databases that WILL BE monitored during auto-discoverypg_exporter_connect_timeout:200# pg_exporter connect timeout in ms, 200 by defaultpg_exporter_options:''# overwrite extra options for pg_exporterpgbouncer_exporter_enabled:true# enable pgbouncer_exporter on pgsql hosts?pgbouncer_exporter_port:9631# pgbouncer_exporter listen port, 9631 by defaultpgbouncer_exporter_url:''# overwrite auto-generate pgbouncer dsn if specifiedpgbouncer_exporter_options:''# overwrite extra options for pgbouncer_exporterpgbackrest_exporter_enabled:true# enable pgbackrest_exporter on pgsql hosts?pgbackrest_exporter_port:9854# pgbackrest_exporter listen port, 9854 by defaultpgbackrest_exporter_options:''# overwrite extra options for pgbackrest_exporter
pg_exporter_enabled
name: pg_exporter_enabled, type: bool, level: C
enable pg_exporter on pgsql hosts?
default value is true, if you don’t want to install pg_exporter, set it to false.
pg_exporter_config
name: pg_exporter_config, type: string, level: C
pg_exporter configuration file name, used by pg_exporter & pgbouncer_exporter
default values: pg_exporter.yml, if you want to use a custom configuration file, you can specify its relative path here.
Your config file should be placed in files/<filename>.yml. For example, if you want to monitor a remote PolarDB instance, you can use the sample config: files/polar_exporter.yml.
pg_exporter_cache_ttls
name: pg_exporter_cache_ttls, type: string, level: C
pg_exporter collector ttl stage in seconds, 1,10,60,300 by default
default values: 1,10,60,300, which will use 1s, 10s, 60s, 300s for different metric collectors.
If you want to customize logging options or other pgbouncer_exporter options, you can set it here.
but do not overwrite pgbouncer_exporter_port here.
pgbackrest_exporter_enabled
name: pgbackrest_exporter_enabled, type: bool, level: C
enable pgbackrest_exporter on pgsql hosts? default value is true
If pgbackrest_enabled is false, this parameter will be short-circuited and disabled.
pgbackrest_exporter_port
name: pgbackrest_exporter_port, type: port, level: C
pgbackrest_exporter listen port, 9854 by default
pgbackrest_exporter_options
name: pgbackrest_exporter_options, type: arg, level: C
extra cli args for pgbackrest_exporter, default value is empty string "".
PG_REMOVE
These flags control the pgsql-rm.yml
playbook and match roles/pg_remove/defaults/main.yml in v3.7.0.
pg_safeguard:false# abort removal when explicitly enabledpg_rm_data:true# remove postgres data during removalpg_rm_backup:true# remove primary pgBackRest backup during removalpg_rm_pkg:true# uninstall postgres packages during removal
pg_safeguard
name: pg_safeguard, type: bool, level: G/C/A
When true, the pgsql-rm.yml playbook aborts before changing the cluster.
The v3.7.0 default is false.
pg_rm_data
name: pg_rm_data, type: bool, level: G/C/A
Remove PostgreSQL data during removal. The default is true; set it to
false to preserve the data directories.
pg_rm_backup
name: pg_rm_backup, type: bool, level: G/C/A
Remove the pgBackRest repository when removing a primary instance. The
default is true; set it to false to preserve backups.
pg_rm_pkg
name: pg_rm_pkg, type: bool, level: G/C/A
Uninstall PostgreSQL and extension packages during removal. The v3.7.0
role default is true; set it to false to keep installed packages.
11.4 - Administration
run administrative tasks
How to maintain an existing PostgreSQL cluster with Pigsty?
pb info # print pgbackrest repo infopg-backup # make a backup, incr, or full backup if necessarypg-backup full # make a full backuppg-backup diff # make a differential backuppg-backup incr # make a incremental backup./pgsql-pitr.yml -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00" }}'./pgsql-pitr.yml -e '{"pg_pitr": { "name": "shit_incoming" }}'./pgsql-pitr.yml -e '{"pg_pitr": { "xid": "250000", exclusive: true }}'./pgsql-pitr.yml -e '{"pg_pitr": { "lsn": "0/4001C80", timeline: "1" }}'
To create a new database user on the existing Postgres cluster, add database definition to all.children.<cls>.pg_databases, then create the database as follows:
Note: If the database has specified an owner, the user should already exist, or you’ll have to Create User first.
Reload Service
Services are exposed access point served by HAProxy.
This task is used when cluster membership has changed, e.g., append/remove replicas, switchover/failover / exposing new service or updating existing service’s config (e.g., LB Weight)
To create new services or reload existing services on entire proxy cluster or specific instances:
It will remove instance <ip> from cluster <cls>. Cluster services will be reloaded to kick the removed instance from load balancer.
Remove Cluster
To remove the entire Postgres cluster, just run:
bin/pgsql-rm <cls> # ./pgsql-rm.yml -l <cls>
Switchover
You can perform a PostgreSQL cluster switchover with patroni cmd.
pg switchover <cls> # interactive mode, you can skip that with following optionspg switchover --leader pg-test-1 --candidate=pg-test-2 --scheduled=now --force pg-test
Backup Cluster
To create a backup with pgBackRest, run as local dbsu:
pg-backup # make a postgres base backuppg-backup full # make a full backuppg-backup diff # make a differential backuppg-backup incr # make a incremental backuppb info # check backup information
The simplest way to achieve a major version upgrade is to create a new cluster with the new version, then migration with logical replication & green/blue deployment.
You can also perform an in-place major upgrade, which is not recommended, especially when certain extensions are installed. But it is possible.
Assume you want to upgrade PostgreSQL 14 to 15, you have to add packages to yum/apt repo, and guarantee the extensions have the exact same version too.
By default, Pigsty uses 25% of memory as PostgreSQL shared buffers, leaving the remaining 75% for the operating system cache.
By default, if users don’t set a pg_max_conn maximum connection count, Pigsty will use default values according to these rules:
oltp: 500 (pgbouncer) / 1000 (postgres)
crit: 500 (pgbouncer) / 1000 (postgres)
tiny: 300
olap: 300
For OLTP and CRIT templates, if services point directly to the PostgreSQL database instead of the pgbouncer connection pool, maximum connections double to 1000.
After determining maximum connections, work_mem is calculated based on shared memory amount / maximum connections, constrained within a 64MB ~ 1GB range.
{% if pg_max_conn != 'auto' and pg_max_conn|int >= 20 %}{% set pg_max_connections = pg_max_conn|int %}{% else %}{% if pg_default_service_dest|default('postgres') == 'pgbouncer' %}{% set pg_max_connections = 500 %}{% else %}{% set pg_max_connections = 1000 %}{% endif %}{% endif %}{% set pg_max_prepared_transactions = pg_max_connections if 'citus' in pg_libs else 0 %}{% set pg_max_locks_per_transaction = (2 * pg_max_connections)|int if 'citus' in pg_libs or 'timescaledb' in pg_libs else pg_max_connections %}{% set pg_shared_buffers = (node_mem_mb|int * pg_shared_buffer_ratio|float) | round(0, 'ceil') | int %}{% set pg_maintenance_mem = (pg_shared_buffers|int * 0.25)|round(0, 'ceil')|int %}{% set pg_effective_cache_size = node_mem_mb|int - pg_shared_buffers|int %}{% set pg_workmem = ([ ([ (pg_shared_buffers / pg_max_connections)|round(0,'floor')|int , 64 ])|max|int , 1024])|min|int %}
CPU Parameters
In PostgreSQL, there are 4 important parameters related to parallel queries. Pigsty automatically optimizes these parameters based on the current system’s CPU core count.
Across all strategies, the total parallel process count (total budget) is typically set to CPU cores + 8, with a minimum of 16, thus reserving sufficient background worker capacity for logical replication and extensions. OLAP and TINY templates vary slightly based on scenario.
OLTP
Configuration Logic
Range Constraints
max_worker_processes
max(100% CPU + 8, 16)
Cores + 4, minimum 12
max_parallel_workers
max(ceil(50% CPU), 2)
1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workers
max(ceil(33% CPU), 2)
1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather
min(max(ceil(20% CPU), 2),8)
1/5 CPU rounded down, minimum 2, maximum 8
OLAP
Configuration Logic
Range Constraints
max_worker_processes
max(100% CPU + 12, 20)
Cores + 12, minimum 20
max_parallel_workers
max(ceil(80% CPU, 2))
4/5 CPU rounded up, minimum 2
max_parallel_maintenance_workers
max(ceil(33% CPU), 2)
1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather
max(floor(50% CPU), 2)
1/2 CPU rounded up, minimum 2
CRIT
Configuration Logic
Range Constraints
max_worker_processes
max(100% CPU + 8, 16)
Cores + 8, minimum 16
max_parallel_workers
max(ceil(50% CPU), 2)
1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workers
max(ceil(33% CPU), 2)
1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather
0, enable as needed
TINY
Configuration Logic
Range Constraints
max_worker_processes
max(100% CPU + 4, 12)
Cores + 4, minimum 12
max_parallel_workers
max(ceil(50% CPU) 1)
50% CPU rounded down, minimum 1
max_parallel_maintenance_workers
max(ceil(33% CPU), 1)
33% CPU rounded down, minimum 1
max_parallel_workers_per_gather
0, enable as needed
Note that CRIT and TINY templates disable parallel queries directly by setting max_parallel_workers_per_gather = 0.
Users can set this parameter as needed to enable parallel queries.
Both OLTP and CRIT templates set the following additional parameters, doubling parallel query costs to reduce the tendency to use parallel queries:
parallel_setup_cost:2000# double from 100 to increase parallel costparallel_tuple_cost:0.2# double from 0.1 to increase parallel costmin_parallel_table_scan_size:16MB # double from 8MB to increase parallel costmin_parallel_index_scan_size:1024# double from 512 to increase parallel cost
Note that max_worker_processes parameter adjustments only take effect after a restart. Additionally, when a replica’s configuration value for this parameter exceeds the primary’s, the replica cannot start.
This parameter must be adjusted through Patroni configuration management. The parameter is managed by Patroni to ensure consistent primary-replica configuration and prevent new replicas from failing to start during failover.
Storage Parameters
Pigsty automatically detects the total disk space where the /data/postgres main data directory resides and uses it as the basis for specifying the following parameters:
min_wal_size:{{([pg_size_twentieth, 200])|min }}GB # 1/20 disk size, max 200GBmax_wal_size:{{([pg_size_twentieth * 4, 2000])|min }}GB # 2/10 disk size, max 2000GBmax_slot_wal_keep_size:{{([pg_size_twentieth * 6, 3000])|min }}GB # 3/10 disk size, max 3000GBtemp_file_limit:{{([pg_size_twentieth, 200])|min }}GB # 1/20 of disk size, max 200GB
temp_file_limit defaults to 5% of disk space, capped at 200GB maximum.
min_wal_size defaults to 5% of disk space, capped at 200GB maximum.
max_wal_size defaults to 20% of disk space, capped at 2TB maximum.
max_slot_wal_keep_size defaults to 30% of disk space, capped at 3TB maximum.
As a special case, the OLAP template allows 20% for temp_file_limit, capped at 2TB maximum.
11.4.2 - Maintenance
Common system maintenance tasks
Ensuring healthy and stable operation of Pigsty and PostgreSQL clusters requires routine maintenance work.
Regular Monitoring Review
Pigsty provides an out-of-the-box monitoring platform. We recommend reviewing monitoring dashboards daily to track system status.
At minimum, we suggest weekly monitoring reviews, focusing on alert events to proactively avoid most failures and issues.
Here’s a list of predefined alert rules in Pigsty.
Failover Follow-up
Pigsty’s high availability architecture allows PostgreSQL clusters to automatically perform primary-replica switching, meaning operations and DBAs don’t require immediate intervention.
However, users still need to perform follow-up tasks at appropriate times (e.g., next business day), including:
Investigate and confirm failure root cause to prevent recurrence
Optionally restore original primary-replica topology or update configuration manifest to match new state
Refresh load balancer configuration via bin/pgsql-svc to update service routing state
Refresh cluster HBA rules via bin/pgsql-hba to prevent primary-replica specific rule drift
If necessary, remove failed servers with bin/pgsql-rm and expand with new replicas using bin/pgsql-add
Regular online rebuilding of tables and indexes using pg_repack helps maintain optimal PostgreSQL performance.
Pigsty installs and enables this extension by default in all databases, ready for immediate use.
You can check table and index bloat through Pigsty’s PGCAT Database - Table Bloat panel.
Select tables and indexes with high bloat rates (larger tables with >50% bloat) for online reorganization using pg_repack:
pg_repack dbname -t schema.table
Normal reads/writes continue during reorganization, but the switch moment at completion requires an AccessExclusive lock, blocking all access.
For high-throughput operations, schedule during low-traffic periods or maintenance windows. For more details, see: Managing Relation Bloat
VACUUM FREEZE
Freezing expired transaction IDs (VACUUM FREEZE) is a critical PostgreSQL maintenance task preventing transaction ID (XID) exhaustion outages.
While PostgreSQL provides AutoVacuum mechanisms, for high-standard production environments,
we recommend combining automatic and manual approaches, regularly executing database-wide VACUUM FREEZE to ensure XID safety.
11.4.3 - Failure SOP
Common failures and troubleshooting strategies
This document outlines potential failures in PostgreSQL and Pigsty, along with SOPs for diagnosing, handling, and analyzing issues.
Disk Space Exhaustion
Disk space exhaustion is the most common type of failure.
Symptoms
When the disk hosting the database runs out of space, PostgreSQL cannot function properly. You may observe: database logs repeatedly reporting “no space left on device”,
inability to write new data, or PostgreSQL triggering a PANIC and forcing shutdown.
Pigsty includes a NodeFsSpaceFull alert rule that triggers when filesystem available space drops below 10%.
Use the monitoring system’s NODE Instance panel to review FS metric panels for diagnosis.
Diagnosis
You can also log into the database node and use df -h to check usage rates for each mount point, determining which partition is full.
For database nodes, focus on these directories and their sizes to determine which file category is consuming space:
Data directory (/pg/data/base): Stores table and index data files, watch for heavy writes and temporary files
WAL directory (e.g., pg/data/pg_wal): Stores PG WAL, WAL accumulation/replication slot retention are common causes of disk exhaustion
Database log directory (e.g., pg/log): If PG logs aren’t rotated timely and massive errors are written, this can consume significant space
Local backup directory (e.g., data/backups): When using pgBackRest to save backups locally, this can also fill the disk
For Pigsty admin nodes or monitoring nodes, also consider:
Monitoring data: Both Prometheus time-series metrics storage and Loki log storage consume disk space, check retention policies
Object storage data: Pigsty’s integrated MinIO object storage may be used for PG backup storage
After identifying directories consuming the most space, use du -sh <directory> to drill down for specific large files or subdirectories.
Resolution
Disk exhaustion is an emergency requiring immediate action to free space and maintain database operation:
Emergency scenario: When data and system disks aren’t separated, disk exhaustion can prevent shell commands from executing. In this case, delete the /pg/dummy placeholder file to free emergency space for shell command recovery.
After freeing space with above measures, PostgreSQL should resume normal operation. If the database crashed due to pg_wal exhaustion, restart the database service after clearing space and carefully verify data integrity.
Transaction ID Wraparound
PostgreSQL uses 32-bit transaction IDs (XIDs) cyclically. When XIDs are exhausted, “transaction ID wraparound” failure occurs.
Symptoms
Initial symptoms include PGSQL Persist - Age Usage panel age saturation entering the warning zone.
Database logs begin showing: WARNING: database "postgres" must be vacuumed within xxxxxxxx transactions.
If the problem worsens, PostgreSQL enters protection mode: when remaining transaction IDs drop below ~1 million, the database switches to read-only mode; at the limit of ~2.1 billion (2^31), it refuses new transactions and forces server shutdown to prevent data corruption.
Diagnosis
PostgreSQL and Pigsty enable AutoVacuum by default, so this failure usually indicates deeper root causes.
Common causes include: super-aged transactions (SAGE), misconfigured Autovacuum, blocked replication slots, insufficient resources, storage engine/extension bugs, disk corruption.
First identify the database with the oldest age, then use the Pigsty PGCAT Database - Tables panel to check table age distribution.
Review database error logs for clues to identify root causes.
Resolution
Immediate transaction freezing: If the database hasn’t entered read-only protection, immediately execute manual VACUUM FREEZE on affected databases. Start with the most aged tables rather than the entire database to expedite results. As superuser, run VACUUM FREEZE tablename; on tables with highest relfrozenxid, prioritizing tables with oldest XID age. This quickly reclaims significant transaction ID space.
Single-user mode rescue: If the database refuses writes or has crashed for protection, start the database in single-user mode for freeze operations. In single-user mode, run VACUUM FREEZE database_name; to freeze-clean the entire database. Then restart in multi-user mode. This releases wraparound locks and restores write capability. Exercise extreme caution in single-user mode and ensure sufficient transaction ID headroom for freezing.
Standby takeover: In complex scenarios (e.g., hardware issues preventing vacuum completion), consider promoting a read-only standby to primary for a cleaner environment. For example, if the primary has bad blocks preventing vacuum, manually failover to promote the standby as new primary, then perform emergency vacuum freeze. After ensuring the new primary has frozen old transactions, switch load back.
Connection Exhaustion
PostgreSQL has a maximum connection limit (max_connections). When client connections exceed this limit, new connection requests are rejected. Typical symptoms include applications unable to connect with errors like
FATAL: remaining connection slots are reserved for non-replication superuser connections or too many clients already.
This indicates regular connection slots are exhausted, leaving only slots reserved for superusers or replication.
Diagnosis
Connection exhaustion typically results from massive concurrent client requests. You can review current active sessions through PGCAT Instance / PGCAT Database / PGCAT Locks
to determine what queries are filling the system for further action. Pay special attention to numerous Idle in Transaction connections and long-running transactions (and slow queries).
Resolution
Kill queries: For exhaustion blocking business operations, immediately use pg_terminate_backend(pid) for emergency relief.
For connection pool users, adjust pool size parameters and reload to reduce database-level connections.
You can also use pg edit-config to increase max_connections, but this parameter requires database restart to take effect.
etcd Quota Exhaustion
etcd quota exhaustion causes PG high availability control plane failure, preventing configuration changes.
Versions between Pigsty v2.0.0 - v2.5.1 are affected by default.
Diagnosis
Pigsty uses etcd as distributed configuration storage (DCS) for high availability. etcd has a storage quota (default ~2GB).
When etcd storage reaches the quota limit, etcd refuses write operations with error “etcdserver: mvcc: database space exceeded”. In this state, Patroni cannot write heartbeats or update configurations to etcd, causing cluster management failure.
Resolution
Pigsty v2.6.0 adds auto-compaction configuration for deployed etcd. If you only use it for PG high availability leases, regular use cases won’t encounter this issue.
Defective Storage Engines
Currently, TimescaleDB’s experimental Hypercore storage engine has proven defects,
with documented cases of VACUUM failing to reclaim XIDs causing wraparound failures.
Users of this feature should migrate promptly to PostgreSQL native tables or TimescaleDB’s default engine.
Handling accidental deletion of data, tables, and databases
Accidental Data Deletion
For small-batch DELETE operations performed in error, consider using the pg_surgery extension for in-place surgical recovery.
If the deleted data has already been reclaimed by VACUUM, follow the general data loss recovery workflow.
Accidental Object Deletion
When DROP/DELETE operations are performed in error, follow this workflow to determine the recovery approach:
Verify if the data can be recovered through business systems or other data sources. If possible, recover directly from the business side.
Check for delayed replica availability. If available, advance the delayed replica to the point before deletion and query the data for recovery.
If data is confirmed deleted, verify backup coverage for the deletion timepoint. If covered, initiate PITR.
Determine whether to perform in-place PITR rollback on the entire cluster, replay on a new server, or use a replica for replay, then execute the recovery strategy.
Accidental Cluster Deletion
In cases where an entire database cluster is accidentally deleted, such as mistakenly executing the pgsql-rm.yml playbook:
Unless you explicitly specified pg_rm_backup: false beforehand, backups will typically be deleted along with the database cluster.
11.5 - Playbook
control primitives
How to manage PostgreSQL cluster with ansible playbooks
Pigsty has a series of playbooks for PostgreSQL:
pgsql.yml : Init HA PostgreSQL clusters or add new replicas.
pgsql-rm.yml : Remove PostgreSQL cluster, or remove replicas
pgsql-db.yml : Add a new business database to existing PostgreSQL cluster
pgsql-user.yml : Add new business user to existing PostgreSQL cluster
pgsql-pitr.yml : Perform point-in-time recovery on existing PostgreSQL cluster
pgsql-monitor.yml : Monitor remote PostgreSQL instance with local exporters
pgsql-migration.yml : Generate Migration manual & scripts for existing PostgreSQL
Safeguard
If you are afraid of accidentally deleting your PostgreSQL cluster, you can enable the safeguard feature.
pg_backup :# remove backup repo (disable with `pg_rm_backup=false`)pg_data :# remove postgres data (disable with `pg_rm_data=false`)pg_pkg :# uninstall pg packages (disable with `pg_rm_pkg=false`)- pg_ext :# uninstall postgres extensions alone
Some arguments can affect the behavior of this playbook:
# remove pgsql cluster `pg-test`./pgsql-rm.yml # remove all the postgres clusters (VERY DANGEROUS)./pgsql-rm.yml -l pg-test # remove the cluster `pg-test`./pgsql-rm.yml -e pg_safeguard=false# force disable safeguard, run this playbook anyway./pgsql-rm.yml -e pg_rm_data=false# keep the data directory, do not remove it (keep the data)./pgsql-rm.yml -e pg_rm_backup=false# do not purge postgres data by default (keep the backup repo)./pgsql-rm.yml -e pg_rm_pkg=false# do not uninstall postgres packages by default (keep the packages)
There are three identity labels: cls, ins, ip, which will be attached to all metrics & logs. node & haproxy will try to reuse the same identity to provide consistent metrics & logs.
Prometheus monitoring targets are defined in static files under /etc/prometheus/targets/pgsql/. Each instance will have a corresponding file. Take pg-meta-1 as an example:
When the global flag patroni_ssl_enabled is set, the patroni target will be managed as /etc/prometheus/targets/patroni/<ins>.yml because it requires a different scrape endpoint (https).
Prometheus monitoring target will be removed when a cluster is removed by bin/pgsql-rm or pgsql-rm.yml. You can use playbook subtasks, or remove them manually:
bin/pgmon-rm <ins> # remove prometheus targets from all infra nodes
Remote RDS targets are managed as /etc/prometheus/targets/pgrds/<cls>.yml. It will be created by the pgsql-monitor.yml playbook or bin/pgmon-add script.
Monitor Mode
There are three ways to monitor PostgreSQL instances in Pigsty:
Suppose the target DB node can be managed by Pigsty (accessible via ssh and sudo is available). In that case, you can use the pg_exporter task in the pgsql.yml playbook to deploy the monitoring component PG Exporter on the target node in the same manner as a standard deployment.
You can also deploy the connection pool and its monitoring on existing instance nodes using the pgbouncer and pgbouncer_exporter tasks from the same playbook. Additionally, you can deploy host monitoring, load balancing, and log collection components using the node_exporter, haproxy, and promtail tasks from the node.yml playbook, achieving a similar user experience with the native Pigsty cluster.
The definition method for existing clusters is very similar to the normal clusters managed by Pigsty. Selectively run certain tasks from the pgsql.yml playbook instead of running the entire playbook.
./node.yml -l <cls> -t node_repo,node_pkg # Add YUM sources for INFRA nodes on host nodes and install packages../node.yml -l <cls> -t node_exporter,node_register # Configure host monitoring and add to Prometheus../node.yml -l <cls> -t promtail # Configure host log collection and send to Loki../pgsql.yml -l <cls> -t pg_exporter,pg_register # Configure PostgreSQL monitoring and register with Prometheus/Grafana.
If you can only access the target database via PGURL (database connection string), you can refer to the instructions here for configuration. In this mode, Pigsty deploys the corresponding PG Exporter on the INFRA node to fetch metrics from the remote database, as shown below:
The monitoring system will no longer have host/pooler/load balancer metrics. But the PostgreSQL metrics & catalog info are still available. Pigsty has two dedicated dashboards for that: PGRDS Cluster and PGRDS Instance. Overview and Database level dashboards are reused. Since Pigsty cannot manage your RDS, you have to setup monitor on the target database in advance.
Below, we use a sandbox environment as an example: now we assume that the pg-meta cluster is an RDS instance pg-foo-1 to be monitored, and the pg-test cluster is an RDS cluster pg-bar to be monitored:
Create monitoring schemas, users, and permissions on the target. Refer to Monitor Setup for details.
Declare the cluster in the configuration list. For example, suppose we want to monitor the “remote” pg-meta & pg-test clusters:
infra:# Infra cluster for proxies, monitoring, alerts, etc.hosts:{10.10.10.10:{infra_seq:1}}vars:# Install pg_exporter on 'infra' group for remote postgres RDSpg_exporters:# List all remote instances here, assign a unique unused local port for k20001:{pg_cluster:pg-foo, pg_seq:1, pg_host:10.10.10.10 , pg_databases:[{name:meta }] }# Register meta database as Grafana data source20002:{pg_cluster:pg-bar, pg_seq:1, pg_host:10.10.10.11 , pg_port:5432}# Several different connection string concatenation methods20003:{pg_cluster:pg-bar, pg_seq:2, pg_host:10.10.10.12 , pg_exporter_url:'postgres://dbuser_monitor:[email protected]:5432/postgres?sslmode=disable'}20004:{pg_cluster:pg-bar, pg_seq:3, pg_host:10.10.10.13 , pg_monitor_username:dbuser_monitor, pg_monitor_password:DBUser.Monitor }
The databases listed in the pg_databases field will be registered in Grafana as a PostgreSQL data source, providing data support for the PGCAT monitoring panel. If you don’t want to use PGCAT and register the database in Grafana, set pg_databases to an empty array or leave it blank.
Execute the command to add monitoring: bin/pgmon-add <clsname>
bin/pgmon-add pg-foo # Bring the pg-foo cluster into monitoringbin/pgmon-add pg-bar # Bring the pg-bar cluster into monitoring
To remove a remote cluster from monitoring, use bin/pgmon-rm <clsname>
bin/pgmon-rm pg-foo # Remove pg-foo from Pigsty monitoringbin/pgmon-rm pg-bar # Remove pg-bar from Pigsty monitoring
You can use more parameters to override the default pg_exporter options. Here is an example for monitoring Aliyun RDS and PolarDB with Pigsty:
Monitor Setup
When you want to monitor existing instances, whether it’s RDS or a self-built PostgreSQL instance, you need to make some configurations on the target database so that Pigsty can access them.
To bring an external existing PostgreSQL instance into monitoring, you need a connection string that can access that instance/cluster. Any accessible connection string (business user, superuser) can be used, but we recommend using a dedicated monitoring user to avoid permission leaks.
Monitor User: The default username used is dbuser_monitor. This user belongs to the pg_monitor group, or ensure it has the necessary view permissions.
Monitor HBA: Default password is DBUser.Monitor. You need to ensure that the HBA policy allows the monitoring user to access the database from the infra nodes.
Monitor Schema: It’s optional but recommended to create a dedicate schema monitor for monitoring views and extensions.
Monitor Extension: It is strongly recommended to enable the built-in extension pg_stat_statements.
Monitor View: Monitoring views are optional but can provide additional metrics. Which is recommended.
Monitor User
Create a monitor user on the target database cluster. For example, dbuser_monitor is used by default in Pigsty.
CREATEUSERdbuser_monitor;-- create the monitor user
COMMENTONROLEdbuser_monitorIS'system monitor user';-- comment the monitor user
GRANTpg_monitorTOdbuser_monitor;-- grant system role pg_monitor to monitor user
ALTERUSERdbuser_monitorPASSWORD'DBUser.Monitor';-- set password for monitor user
ALTERUSERdbuser_monitorSETlog_min_duration_statement=1000;-- set this to avoid log flooding
ALTERUSERdbuser_monitorSETsearch_path=monitor,public;-- set this to avoid pg_stat_statements extension not working
You also need to configure pg_hba.conf to allow monitoring user access from infra/admin nodes.
# allow local role monitor with passwordlocal all dbuser_monitor md5host all dbuser_monitor 127.0.0.1/32 md5host all dbuser_monitor <admin_ip>/32 md5host all dbuser_monitor <infra_ip>/32 md5
If your RDS does not support the RAW HBA format, add admin/infra node IP to the whitelist.
Monitor Schema
Monitor schema is optional, but we strongly recommend creating one.
CREATESCHEMAIFNOTEXISTSmonitor;-- create dedicate monitor schema
GRANTUSAGEONSCHEMAmonitorTOdbuser_monitor;-- allow monitor user to use this schema
Monitor Extension
Monitor extension is optional, but we strongly recommend enabling pg_stat_statements extension.
Note that this extension must be listed in shared_preload_libraries to take effect, and changing this parameter requires a database restart.
You should create this extension inside the admin database: postgres. If your RDS does not grant CREATE on the database postgres. You can create that extension in the default public schema:
As long as your monitor user can access pg_stat_statements view without schema qualification, it should be fine.
Monitor View
It’s recommended to create the monitor views in all databases that need to be monitored.
Monitor Schema & View Definition
----------------------------------------------------------------------
-- Table bloat estimate : monitor.pg_table_bloat
----------------------------------------------------------------------
DROPVIEWIFEXISTSmonitor.pg_table_bloatCASCADE;CREATEORREPLACEVIEWmonitor.pg_table_bloatASSELECTCURRENT_CATALOGASdatname,nspname,relname,tblid,bs*tblpagesASsize,CASEWHENtblpages-est_tblpages_ff>0THEN(tblpages-est_tblpages_ff)/tblpages::FLOATELSE0ENDASratioFROM(SELECTceil(reltuples/((bs-page_hdr)*fillfactor/(tpl_size*100)))+ceil(toasttuples/4)ASest_tblpages_ff,tblpages,fillfactor,bs,tblid,nspname,relname,is_naFROM(SELECT(4+tpl_hdr_size+tpl_data_size+(2*ma)-CASEWHENtpl_hdr_size%ma=0THENmaELSEtpl_hdr_size%maEND-CASEWHENceil(tpl_data_size)::INT%ma=0THENmaELSEceil(tpl_data_size)::INT%maEND)AStpl_size,(heappages+toastpages)AStblpages,heappages,toastpages,reltuples,toasttuples,bs,page_hdr,tblid,nspname,relname,fillfactor,is_naFROM(SELECTtbl.oidAStblid,ns.nspname,tbl.relname,tbl.reltuples,tbl.relpagesASheappages,coalesce(toast.relpages,0)AStoastpages,coalesce(toast.reltuples,0)AStoasttuples,coalesce(substring(array_to_string(tbl.reloptions,' ')FROM'fillfactor=([0-9]+)')::smallint,100)ASfillfactor,current_setting('block_size')::numericASbs,CASEWHENversion()~'mingw32'ORversion()~'64-bit|x86_64|ppc64|ia64|amd64'THEN8ELSE4ENDASma,24ASpage_hdr,23+CASEWHENMAX(coalesce(s.null_frac,0))>0THEN(7+count(s.attname))/8ELSE0::intEND+CASEWHENbool_or(att.attname='oid'andatt.attnum<0)THEN4ELSE0ENDAStpl_hdr_size,sum((1-coalesce(s.null_frac,0))*coalesce(s.avg_width,0))AStpl_data_size,bool_or(att.atttypid='pg_catalog.name'::regtype)ORsum(CASEWHENatt.attnum>0THEN1ELSE0END)<>count(s.attname)ASis_naFROMpg_attributeASattJOINpg_classAStblONatt.attrelid=tbl.oidJOINpg_namespaceASnsONns.oid=tbl.relnamespaceLEFTJOINpg_statsASsONs.schemaname=ns.nspnameANDs.tablename=tbl.relnameANDs.inherited=falseANDs.attname=att.attnameLEFTJOINpg_classAStoastONtbl.reltoastrelid=toast.oidWHERENOTatt.attisdroppedANDtbl.relkind='r'ANDnspnameNOTIN('pg_catalog','information_schema')GROUPBY1,2,3,4,5,6,7,8,9,10)ASs)ASs2)ASs3WHERENOTis_na;COMMENTONVIEWmonitor.pg_table_bloatIS'postgres table bloat estimate';GRANTSELECTONmonitor.pg_table_bloatTOpg_monitor;----------------------------------------------------------------------
-- Index bloat estimate : monitor.pg_index_bloat
----------------------------------------------------------------------
DROPVIEWIFEXISTSmonitor.pg_index_bloatCASCADE;CREATEORREPLACEVIEWmonitor.pg_index_bloatASSELECTCURRENT_CATALOGASdatname,nspname,idxnameASrelname,tblid,idxid,relpages::BIGINT*bsASsize,COALESCE((relpages-(reltuples*(6+ma-(CASEWHENindex_tuple_hdr%ma=0THENmaELSEindex_tuple_hdr%maEND)+nulldatawidth+ma-(CASEWHENnulldatawidth%ma=0THENmaELSEnulldatawidth%maEND))/(bs-pagehdr)::FLOAT+1)),0)/relpages::FLOATASratioFROM(SELECTnspname,idxname,indrelidAStblid,indexrelidASidxid,reltuples,relpages,current_setting('block_size')::INTEGERASbs,(CASEWHENversion()~'mingw32'ORversion()~'64-bit|x86_64|ppc64|ia64|amd64'THEN8ELSE4END)ASma,24ASpagehdr,(CASEWHENmax(COALESCE(pg_stats.null_frac,0))=0THEN2ELSE6END)ASindex_tuple_hdr,sum((1.0-COALESCE(pg_stats.null_frac,0.0))*COALESCE(pg_stats.avg_width,1024))::INTEGERASnulldatawidthFROMpg_attributeJOIN(SELECTpg_namespace.nspname,ic.relnameASidxname,ic.reltuples,ic.relpages,pg_index.indrelid,pg_index.indexrelid,tc.relnameAStablename,regexp_split_to_table(pg_index.indkey::TEXT,' ')::INTEGERASattnum,pg_index.indexrelidASindex_oidFROMpg_indexJOINpg_classicONpg_index.indexrelid=ic.oidJOINpg_classtcONpg_index.indrelid=tc.oidJOINpg_namespaceONpg_namespace.oid=ic.relnamespaceJOINpg_amONic.relam=pg_am.oidWHEREpg_am.amname='btree'ANDic.relpages>0ANDnspnameNOTIN('pg_catalog','information_schema'))ind_attsONpg_attribute.attrelid=ind_atts.indexrelidANDpg_attribute.attnum=ind_atts.attnumJOINpg_statsONpg_stats.schemaname=ind_atts.nspnameAND((pg_stats.tablename=ind_atts.tablenameANDpg_stats.attname=pg_get_indexdef(pg_attribute.attrelid,pg_attribute.attnum,TRUE))OR(pg_stats.tablename=ind_atts.idxnameANDpg_stats.attname=pg_attribute.attname))WHEREpg_attribute.attnum>0GROUPBY1,2,3,4,5,6)est;COMMENTONVIEWmonitor.pg_index_bloatIS'postgres index bloat estimate (btree-only)';GRANTSELECTONmonitor.pg_index_bloatTOpg_monitor;----------------------------------------------------------------------
-- Relation Bloat : monitor.pg_bloat
----------------------------------------------------------------------
DROPVIEWIFEXISTSmonitor.pg_bloatCASCADE;CREATEORREPLACEVIEWmonitor.pg_bloatASSELECTcoalesce(ib.datname,tb.datname)ASdatname,coalesce(ib.nspname,tb.nspname)ASnspname,coalesce(ib.tblid,tb.tblid)AStblid,coalesce(tb.nspname||'.'||tb.relname,ib.nspname||'.'||ib.tblid::RegClass)AStblname,tb.sizeAStbl_size,CASEWHENtb.ratio<0THEN0ELSEround(tb.ratio::NUMERIC,6)ENDAStbl_ratio,(tb.size*(CASEWHENtb.ratio<0THEN0ELSEtb.ratio::NUMERICEND))::BIGINTAStbl_wasted,ib.idxid,ib.nspname||'.'||ib.relnameASidxname,ib.sizeASidx_size,CASEWHENib.ratio<0THEN0ELSEround(ib.ratio::NUMERIC,5)ENDASidx_ratio,(ib.size*(CASEWHENib.ratio<0THEN0ELSEib.ratio::NUMERICEND))::BIGINTASidx_wastedFROMmonitor.pg_index_bloatibFULLOUTERJOINmonitor.pg_table_bloattbONib.tblid=tb.tblid;COMMENTONVIEWmonitor.pg_bloatIS'postgres relation bloat detail';GRANTSELECTONmonitor.pg_bloatTOpg_monitor;----------------------------------------------------------------------
-- monitor.pg_index_bloat_human
----------------------------------------------------------------------
DROPVIEWIFEXISTSmonitor.pg_index_bloat_humanCASCADE;CREATEORREPLACEVIEWmonitor.pg_index_bloat_humanASSELECTidxnameASname,tblname,idx_wastedASwasted,pg_size_pretty(idx_size)ASidx_size,round(100*idx_ratio::NUMERIC,2)ASidx_ratio,pg_size_pretty(idx_wasted)ASidx_wasted,pg_size_pretty(tbl_size)AStbl_size,round(100*tbl_ratio::NUMERIC,2)AStbl_ratio,pg_size_pretty(tbl_wasted)AStbl_wastedFROMmonitor.pg_bloatWHEREidxnameISNOTNULL;COMMENTONVIEWmonitor.pg_index_bloat_humanIS'postgres index bloat info in human-readable format';GRANTSELECTONmonitor.pg_index_bloat_humanTOpg_monitor;----------------------------------------------------------------------
-- monitor.pg_table_bloat_human
----------------------------------------------------------------------
DROPVIEWIFEXISTSmonitor.pg_table_bloat_humanCASCADE;CREATEORREPLACEVIEWmonitor.pg_table_bloat_humanASSELECTtblnameASname,idx_wasted+tbl_wastedASwasted,pg_size_pretty(idx_wasted+tbl_wasted)ASall_wasted,pg_size_pretty(tbl_wasted)AStbl_wasted,pg_size_pretty(tbl_size)AStbl_size,tbl_ratio,pg_size_pretty(idx_wasted)ASidx_wasted,pg_size_pretty(idx_size)ASidx_size,round(idx_wasted::NUMERIC*100.0/idx_size,2)ASidx_ratioFROM(SELECTdatname,nspname,tblname,coalesce(max(tbl_wasted),0)AStbl_wasted,coalesce(max(tbl_size),1)AStbl_size,round(100*coalesce(max(tbl_ratio),0)::NUMERIC,2)AStbl_ratio,coalesce(sum(idx_wasted),0)ASidx_wasted,coalesce(sum(idx_size),1)ASidx_sizeFROMmonitor.pg_bloatWHEREtblnameISNOTNULLGROUPBY1,2,3)d;COMMENTONVIEWmonitor.pg_table_bloat_humanIS'postgres table bloat info in human-readable format';GRANTSELECTONmonitor.pg_table_bloat_humanTOpg_monitor;----------------------------------------------------------------------
-- Activity Overview: monitor.pg_session
----------------------------------------------------------------------
DROPVIEWIFEXISTSmonitor.pg_sessionCASCADE;CREATEORREPLACEVIEWmonitor.pg_sessionASSELECTcoalesce(datname,'all')ASdatname,numbackends,active,idle,ixact,max_duration,max_tx_duration,max_conn_durationFROM(SELECTdatname,count(*)ASnumbackends,count(*)FILTER(WHEREstate='active')ASactive,count(*)FILTER(WHEREstate='idle')ASidle,count(*)FILTER(WHEREstate='idle in transaction'ORstate='idle in transaction (aborted)')ASixact,max(extract(epochfromnow()-state_change))FILTER(WHEREstate='active')ASmax_duration,max(extract(epochfromnow()-xact_start))ASmax_tx_duration,max(extract(epochfromnow()-backend_start))ASmax_conn_durationFROMpg_stat_activityWHEREbackend_type='client backend'ANDpid<>pg_backend_pid()GROUPBYROLLUP(1)ORDERBY1NULLSFIRST)t;COMMENTONVIEWmonitor.pg_sessionIS'postgres activity group by session';GRANTSELECTONmonitor.pg_sessionTOpg_monitor;----------------------------------------------------------------------
-- Sequential Scan: monitor.pg_seq_scan
----------------------------------------------------------------------
DROPVIEWIFEXISTSmonitor.pg_seq_scanCASCADE;CREATEORREPLACEVIEWmonitor.pg_seq_scanASSELECTschemanameASnspname,relname,seq_scan,seq_tup_read,seq_tup_read/seq_scanASseq_tup_avg,idx_scan,n_live_tup+n_dead_tupAStuples,round(n_live_tup*100.0::NUMERIC/(n_live_tup+n_dead_tup),2)ASlive_ratioFROMpg_stat_user_tablesWHEREseq_scan>0and(n_live_tup+n_dead_tup)>0ORDERBYseq_scanDESC;COMMENTONVIEWmonitor.pg_seq_scanIS'table that have seq scan';GRANTSELECTONmonitor.pg_seq_scanTOpg_monitor;
11.7 - FAQ
frequently asked questions
ABORT due to postgres exists
This happens when you run pgsql.yml on a node with postgres running.
If there’s a running postgres instance, you can explicitly remove it with pgsql-rm.yml playbook:
./pgsql-rm.yml -l <cls_to_remove> # remove the cluster 'cls_to_remove'
ABORT due to pg_safeguard enabled
Disable pg_safeguard to remove the Postgres instance.
If pg_safeguard is enabled, you cannot remove the running pgsql instance with bin/pgsql-rm and pgsql-rm.yml playbook.
To disable pg_safeguard, you can set pg_safeguard to false in the inventory or pass -e pg_safeguard=false as cli arg to the playbook:
./pgsql-rm.yml -e pg_safeguard=false -l <cls_to_remove> # force override pg_safeguard
Fail to wait for postgres/patroni primary
There are several possible reasons for this error, and you need to check the system logs to determine the actual cause.
This usually happens when the cluster is misconfigured, or the previous primary is improperly removed. (e.g., trash metadata in DCS with the same cluster name).
You must check /pg/log/* to find the reason.
To delete trash meta from etcd, you can use etcdctl del --prefix /pg/<cls>, do with caution!
1: Misconfiguration. Identify the incorrect parameters, modify them, and apply the changes.
2: Another cluster with the same cls name already exists in the deployment
3: The previous cluster on the node, or previous cluster with same name was not correctly removed.
To remove obsolete cluster metadata, you can use etcdctl del --prefix /pg/<cls> to manually delete the residual data.
4: The RPM packages related to your PostgreSQL or node were not successfully installed.
5: Your Watchdog kernel module was not correctly enabled or loaded, but required.
6: The locale or ctype specified pg_lc_collate and pg_lc_ctype does not exist in OS
Feel free to submit an issue or seek help from the community.
Fail to wait for postgres/patroni replica
Failed Immediately: Usually, this happens because of misconfiguration, network issues, broken DCS metadata, etc…, you have to inspect /pg/log to find out the actual reason.
Failed After a While: This may be due to source instance data corruption. Check PGSQL FAQ: How to create replicas when data is corrupted?
Timeout: If the wait for postgres replica task takes 30min or more and fails due to timeout, This is common for a huge cluster (e.g., 1TB+, which may take hours to create a replica). In this case, the underlying creating replica procedure is still proceeding. You can check cluster status with pg list <cls> and wait until the replica catches up with the primary. Then continue the following tasks:
To install PostgreSQL 13 ~ 17, you have to set pg_version to 13, 14, 15, 16, or 17 in the inventory. (usually at cluster level)
pg_version:17# install pg 17 in this template
How enable hugepage for PostgreSQL?
use node_hugepage_count and node_hugepage_ratio or /pg/bin/pg-tune-hugepage
If you plan to enable hugepage, consider using node_hugepage_count and node_hugepage_ratio and apply with ./node.yml -t node_tune .
It’s good to allocate enough hugepage before postgres start, and use pg_tune_hugepage to shrink them later.
If your postgres is already running, you can use /pg/bin/pg-tune-hugepage to enable hugepage on the fly. Note that this only works on PostgreSQL 15+
sync;echo3 > /proc/sys/vm/drop_caches # drop system cache (ready for performance impact)sudo /pg/bin/pg-tune-hugepage # write nr_hugepages to /etc/sysctl.d/hugepage.confpg restart <cls> # restart postgres to use hugepage
How to guarantee zero data loss during failover?
Use crit.yml template, or setting pg_rpo to 0, or config cluster with synchronous mode.
The pg_dummy_filesize is set to 64MB by default. Consider increasing it to 8GB or larger in the production environment.
It will be placed on /pg/dummy same disk as the PGSQL main data disk. You can remove that file to free some emergency space. At least you can run some shell scripts on that node.
How to create replicas when data is corrupted?
Disable clonefrom on bad instances and reload patroni config.
Pigsty sets the cloneform: true tag on all instances’ patroni config, which marks the instance available for cloning replica.
If this instance has corrupt data files, you can set clonefrom: false to avoid pulling data from the evil instance. To do so:
bin/pgmon-rm <ins> # shortcut for removing prometheus targets of pgsql instance 'ins'
11.8 - User Role
In this context, User refers to logical objects created by SQL CREATE USER / ROLE
You can manage PostgreSQL users and roles with Pigsty, in an IaC manner.
Define User
You can define roles/users with the following parameters, they are both arrays consisting of user objects:
pg_users : Define business users & roles at cluster level (Cluster Definition)
pg_default_roles : Define system-wide roles & global users (Global Defaults)
The former defines global roles and users shared across the entire environment, while the latter defines business roles and users specific to a single cluster.
Here are some examples of user definitions:
pg-meta:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-metapg_databases:- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer for meta database }- {name:dbuser_grafana ,password:DBUser.Grafana ,pgbouncer:true ,roles:[dbrole_admin] ,comment:admin user for grafana database }- {name:dbuser_bytebase ,password:DBUser.Bytebase ,pgbouncer:true ,roles:[dbrole_admin] ,comment:admin user for bytebase database }- {name:dbuser_kong ,password:DBUser.Kong ,pgbouncer:true ,roles:[dbrole_admin] ,comment:admin user for kong api gateway }- {name:dbuser_gitea ,password:DBUser.Gitea ,pgbouncer:true ,roles:[dbrole_admin] ,comment:admin user for gitea service }- {name:dbuser_wiki ,password:DBUser.Wiki ,pgbouncer:true ,roles:[dbrole_admin] ,comment:admin user for wiki.js service }- {name:dbuser_noco ,password:DBUser.Noco ,pgbouncer:true ,roles:[dbrole_admin] ,comment:admin user for nocodb service }
User Attributes
You can customize users with more attributes, the full example is as follows:
- name:dbuser_meta # REQUIRED, `name` is the only mandatory field of a user definitionpassword:DBUser.Meta # optional, password, can be a scram-sha-256 hash string or plain textlogin:true# optional, can log in, true by default (new biz ROLE should be false)superuser:false# optional, is superuser? false by defaultcreatedb:false# optional, can create database? false by defaultcreaterole:false# optional, can create role? false by defaultinherit:true# optional, can this role use inherited privileges? true by defaultreplication:false# optional, can this role do replication? false by defaultbypassrls:false# optional, can this role bypass row level security? false by defaultpgbouncer:true# optional, add this user to pgbouncer user-list? false by default (production user should be true explicitly)connlimit:-1# optional, user connection limit, default -1 disable limitexpire_in:3650# optional, now + n days when this role is expired (OVERWRITE expire_at)expire_at:'2030-12-31'# optional, YYYY-MM-DD 'timestamp' when this role is expired (OVERWRITTEN by expire_in)comment:pigsty admin user # optional, comment string for this user/roleroles:[dbrole_admin] # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}parameters:{}# optional, role level parameters with `ALTER ROLE SET`pool_mode:transaction # optional, pgbouncer pool mode at user level, transaction by defaultpool_connlimit:-1# optional, max database connections at user level, default -1 disable limitsearch_path:public # key value config parameters, according to postgresql documentation (e.g: use pigsty as default search_path)
The only required field is name, which should be a valid & unique username in PostgreSQL.
Roles don’t need a password, while it could be necessary for a login-able user.
The password can be plain text or a scram-sha-256 / md5 hash string.
User / Role definition order matters, pg_default_roles first, pg_users later, in sequence order.
Make sure role / group definition is ahead of its members.
Role Attributes: login, superuser, createdb, createrole, inherit, replication, bypassrls
pgbouncer is disabled by default. Set it to true explicitly to enable it in pgbouncer.
ACL System
Pigsty has a battery-included ACL system, which can be easily used by assigning roles to users:
dbrole_readonly : The role for global read-only access
dbrole_readwrite : The role for global read-write access
dbrole_admin : The role for object creation
dbrole_offline : The role for restricted read-only access (offline instance)
If you wish to re-design your ACL system, check the following parameters and SQL templates.
Users and roles defined in pg_default_roles and pg_users will be automatically created one by one during module installation.
It only runs on cluster leader, the primary instance.
To create users on an existing cluster,
add new user/role definitions to all.children.<cls>.pg_users, and create the database with the bin/pgsql-user util or pgsql-user.yml playbook:
bin/pgsql-user <cls> <dbname> # the bin util scriptbin/pgsql-user pg-meta dbuser_meta # example: create dbuser_meta user in pg-meta cluster
./pgsql-user.yml -l <cls> -e username=<dbname> # the actual playbook./pgsql-user.yml -l pg-meta -e username=meta # example: create dbuser_meta user in pg-meta cluster
Create user is an idempotent operation, meaning it can be run multiple times safely.
Create user / role with Pigsty
Pigsty will manage the pgbouncer userlist, so please create business databases with the Pigsty playbook/utils.
Check create user SOP for details.
If you are not using pgbouncer or able to maintain it by yourself, you can create users in any way you like.
Create owner user before create database
In PostgreSQL, users belong to the database cluster, not a specific database.
If your user is an owner of any databases, make sure the user is created before creating the database.
There are two exceptions: name and roles, which require manual intervention:
Rename user is not supported directly in Pigsty
The username is used as the identity of the user, so if you really want to do that, use the standard SQL:
ALTERUSER"old_name"RENAMETO"new_name";
Membership will NOT be revoked by Pigsty
Note that modifying a user does not delete the user, but modifies user attributes using the ALTER USER command.
It also DOES NOT revoke user permissions and group memberships, and uses the GRANT command to grant new roles.
Check PostgreSQL Docs for more details on ALTER USER.
Delete User
For security reasons, Pigsty does not automatically delete users, even if you remove user definitions from the configuration, Pigsty will not delete existing users.
You need to use the SQL command DROP USER to manually delete users:
DROPUSER"<username>";
If the role you want to delete is a group (has other users belonging to it), you need to first remove other users from the group before deleting the group:
REVOKE"<rolename>"FROM"<other_user>";
If the user you want to delete owns database objects, you need to first change the ownership of these objects to another user before deleting the user:
Pigsty helps manage users in pgbouncer userlist, and keep it in sync with the postgres.
It requires explicitly setting the pgbouncer: true flag in the user definition to be enrolled in the pgbouncer user list.
The system admin user (pg_admin_username) and monitoring user (pg_monitor_username)
will always be added to the pgbouncer user list for administration & monitoring.
Configuration Files
Users in the Pgbouncer connection pool are listed in /etc/pgbouncer/userlist.txt, examples:
The userlist.txt and useropts.txt will be automatically refreshed when you create users
and take effect with systemctl reload pgbouncer, normally without affecting existing connections.
Reload
To reload pgbouncer configuration, you can use the ansible playbook, or systemctl command
./pgsql.yml -t pgbouncer_reload
systemctl reload pgbouncer
Admin
Pgbouncer runs with the same dbsu as PostgreSQL, defaulting to the postgres os user.
You can use the pgb alias to access pgbouncer management functions using dbsu.
postgres
sudo su - postgres
pgb # login to pgbouncer command line interface using admin user
Delete Pgbouncer User
If all database users are managed by Pigsty, you can just regenerate pgbouncer userlist (without the removed user in the list in the config inventory) and reload it:
To manually remove a user from the pgbouncer pool, simply delete the corresponding line from /etc/pgbouncer/userlist.txt and reload pgbouncer:
systemctl reload pgbouncer
Dynamic User Authentication
Note that the pgbouncer_auth_query parameter allows you to use dynamic queries to complete connection pool user authentication, which is a compromise when you don’t want to manage users in the connection pool.
11.9 - Database
In this context, Database refers to the object created by SQL CREATE DATABASE.
A PostgreSQL server can serve multiple databases simultaneously. You can manage them with Pigsty.
Define Database
Business databases are defined by pg_databases, which is a cluster-level parameter.
For example, the default meta database is defined in the pg-meta cluster:
Each database definition is a dict with the following fields:
- name:meta # REQUIRED, `name` is the only mandatory field of a database definitionbaseline:cmdb.sql # optional, database sql baseline path, (relative path among ansible search path, e.g files/)pgbouncer:true# optional, add this database to pgbouncer database list? true by defaultschemas:[pigsty] # optional, additional schemas to be created, array of schema namesextensions:# optional, additional extensions to be installed: array of `{name[,schema]}`- {name:postgis , schema:public }- {name:timescaledb }comment:pigsty meta database # optional, comment string for this databaseowner:postgres # optional, database owner, postgres by defaulttemplate:template1 # optional, which template to use, template1 by defaultencoding:UTF8 # optional, database encoding, UTF8 by default. (MUST same as template database)locale:C # optional, database locale, C by default. (MUST same as template database)lc_collate:C # optional, database collate, C by default. (MUST same as template database)lc_ctype:C # optional, database ctype, C by default. (MUST same as template database)tablespace:pg_default # optional, default tablespace, 'pg_default' by default.allowconn:true# optional, allow connection, true by default. false will disable connect at allrevokeconn:false# optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)register_datasource:true# optional, register this database to grafana datasources? true by defaultconnlimit:-1# optional, database connection limit, default -1 disable limitpool_auth_user:dbuser_meta # optional, all connection to this pgbouncer database will be authenticated by this userpool_mode:transaction # optional, pgbouncer pool mode at database level, default transactionpool_size:64# optional, pgbouncer pool size at database level, default 64pool_size_reserve:32# optional, pgbouncer pool size reserve at database level, default 32pool_size_min:0# optional, pgbouncer pool size min at database level, default 0pool_max_db_conn:100# optional, max database connections at database level, default 100
The only required field is name, which should be a valid and unique database name in PostgreSQL.
Newly created databases are forked from template1 database by default. Which is customized by PG_PROVISION during cluster bootstrap.
Databases defined in pg_databases will be automatically created during module installation.
If you wish to create database on an existing cluster, the bin/pgsql-db util can be used.
Add new database definition to all.children.<cls>.pg_databases, and create that database with:
bin/pgsql-db <cls> <dbname> # the bin util scriptbin/pgsql-db pg-meta meta # example: create meta database in pg-meta cluster
./pgsql-db.yml -l <cls> -e dbname=<dbname> # the actual playbook./pgsql-db.yml -l pg-meta -e dbname=meta # example: create meta database in pg-meta cluster
This playbook is usually idempotent and can be re-run to flush the database definition.
But if you have non-trivial baseline schema (like drop stuff), you should NOT re-run this on existing databases.
Create postgres database with pigsty
Pigsty will manage pgbouncer database list, so please create business databases with the Pigsty playbook/utils.
Check create database SOP for details.
If you are not using pgbouncer or able to maintain it by yourself, you can create databases in any way you like.
Create owner before create database
If your database has a non-trivial owner (dbsu postgres by default), make sure the owner user exists before creating the database.
In short, always create the users before creating databases.
Pgbouncer Database
Pgbouncer is enabled by default and serves as connection pool middleware.
Pigsty will add all databases in pg_databases to the pgbouncer database list by default.
You can disable the pgbouncer proxy for a specific database by setting pgbouncer: false in the database definition.
The Pgbouncer database list will be updated when create database with Pigsty util & playbook.
Databases are listed in /etc/pgbouncer/database.txt, with extra database-level parameters:
When you create databases, the Pgbouncer database list definition file will be refreshed and take effect through online configuration reload, without affecting existing connections.
To access pgbouncer admin functionality, you can use the pgb alias as dbsu (postgres).
Check pgbouncer usage for available commands:
postgres
sudo su - postgres # switch to the postgres dbsupgb # access the pgbouncer admin virtual database
There’s a util function defined in /etc/profile.d/pg-alias.sh, allowing you to reroute pgbouncer database traffic to a new host quickly, which can be used during zero-downtime migration.
/etc/profile.d/pg-alias.sh
# route pgbouncer traffic to another cluster memberfunction pgb-route(){localip=${1-'\/var\/run\/postgresql'} sed -ie "s/host=[^[:space:]]\+/host=${ip}/g" /etc/pgbouncer/pgbouncer.ini
cat /etc/pgbouncer/pgbouncer.ini
}
11.10 - Service
reliable service access via lb, proxy, pool
Service Implementation
In Pigsty, services are implemented using haproxy on nodes, differentiated by different ports on the host node.
Every node has Haproxy enabled to expose services. From the database perspective, nodes in the cluster may be primary or replicas, but from the service perspective, all nodes are the same. This means even if you access a replica node, as long as you use the correct service port, you can still use the primary’s read-write service. This design seals the complexity: as long as you can access any instance on the PostgreSQL cluster, you can fully access all services.
This design is akin to the NodePort service in Kubernetes. Similarly, in Pigsty, every service includes these two core elements:
Access endpoints exposed via NodePort (port number, from where to access?)
Target instances chosen through Selectors (list of instances, who will handle it?)
The boundary of Pigsty’s service delivery stops at the cluster’s HAProxy. Users can access these load balancers in various ways. Please refer to Access Service.
All services are declared through configuration files. For instance, the default PostgreSQL service is defined by the pg_default_services parameter:
While you can define your extra PostgreSQL services with pg_services @ the global or cluster level.
These two parameters are both arrays of service objects. Each service definition will be rendered as a haproxy config in /etc/haproxy/<svcname>.cfg, check service.cfg for details.
Here is an example of an extra service definition: standby
- name:standby # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standbyport:5435# required, service exposed port (work as kubernetes service node port mode)ip:"*"# optional, service bind ip address, `*` for all ip by defaultselector:"[]"# required, service member selector, use JMESPath to filter inventorydest:default # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by defaultcheck:/sync # optional, health check url path, / by defaultbackup:"[? pg_role == `primary`]"# backup server selectormaxconn:3000# optional, max allowed front-end connectionbalance:roundrobin # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)options:'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'
And it will be translated to a haproxy config file /etc/haproxy/pg-test-standby.conf:
#---------------------------------------------------------------------# service: pg-test-standby @ 10.10.10.11:5435#---------------------------------------------------------------------# service instances 10.10.10.11, 10.10.10.13, 10.10.10.12# service backups 10.10.10.11listen pg-test-standbybind *:5435mode tcpmaxconn 5000balance roundrobinoption httpchkoption http-keep-alivehttp-check send meth OPTIONS uri /sync # <--- true for primary & sync standbyhttp-check expect status 200default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100# serversserver pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup # the primary is used as backup serverserver pg-test-3 10.10.10.13:6432 check port 8008 weight 100server pg-test-2 10.10.10.12:6432 check port 8008 weight 100
Reload Service
When cluster membership has changed, such as append / remove replicas, switchover/failover, or adjust relative weight, You have to reload service to make the changes take effect.
bin/pgsql-svc <cls> [ip...]# reload service for lb cluster or lb instance# ./pgsql.yml -t pg_service # the actual ansible task to reload service
Override Service
You can override the default service configuration in several ways:
Bypass Pgbouncer
When defining a service, if svc.dest='default', this parameter pg_default_service_dest will be used as the default value. pgbouncer is used by default, you can use postgres instead, so the default primary & replica service will bypass pgbouncer and route traffic to postgres directly
If you don’t need connection pooling at all, you can change pg_default_service_dest to postgres, and remove default and offline services.
If you don’t need read-only replicas for online traffic, you can remove replica from pg_default_services too.
For example, this configuration will expose pg cluster primary service on haproxy node group proxy with port 10013.
pg_service_provider:proxy # use load balancer on group `proxy` with port 10013pg_default_services:[{name:primary ,port:10013 ,dest:postgres ,check:/primary ,selector:"[]"}]
It’s user’s responsibility to make sure each delegate service port is unique among the proxy cluster.
Split read & write, route traffic to the right place, and achieve stable & reliable access to the PostgreSQL cluster.
Service is an abstraction to seal the details of the underlying cluster, especially during cluster failover/switchover.
Personal User
Service is meaningless to personal users. You can access the database with raw IP address or whatever method you like.
psql postgres://dbuser_dba:[email protected]/meta # dbsu direct connectpsql postgres://dbuser_meta:[email protected]/meta # default business admin userpsql postgres://dbuser_view:DBUser.View@pg-meta/meta # default read-only user
Service Overview
We utilize a PostgreSQL database cluster based on replication in real-world production environments. Within the cluster, only one instance is the leader (primary) that can accept writes. Other instances (replicas) continuously fetch WAL from the leader to stay synchronized. Additionally, replicas can handle read-only queries and offload the primary in read-heavy, write-light scenarios. Thus, distinguishing between write and read-only requests is a common practice.
Moreover, we pool requests through a connection pooling middleware (Pgbouncer) for high-frequency, short-lived connections to reduce the overhead of connection and backend process creation. And, for scenarios like ETL and change execution, we need to bypass the connection pool and directly access the database servers. Furthermore, high-availability clusters may undergo failover during failures, causing a change in the cluster leadership. Therefore, the RW requests should be re-routed automatically to the new leader.
These varied requirements (read-write separation, pooling vs. direct connection, and client request failover) have led to the abstraction of the service concept.
Typically, a database cluster must provide this basic service:
Read-write service (primary): Can read and write to the database.
For production database clusters, at least these two services should be provided:
Read-write service (primary): Write data: Only carried by the primary.
Read-only service (replica): Read data: Can be carried by replicas, but fallback to the primary if no replicas are available.
Additionally, there might be other services, such as:
Direct access service (default): Allows (admin) users to bypass the connection pool and directly access the database.
Offline replica service (offline): A dedicated replica that doesn’t handle online read traffic, used for ETL and analytical queries.
Synchronous replica service (standby): A read-only service with no replication delay, handled by synchronous standby/primary for read queries.
Delayed replica service (delayed): Accesses older data from the same cluster from a certain time ago, handled by delayed replicas.
Default Service
Pigsty will enable four default services for each PostgreSQL cluster:
Take the default pg-meta cluster as an example, you can access these services in the following ways:
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta # pg-meta-primary : production read/write via primary pgbouncer(6432)psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta # pg-meta-replica : production read-only via replica pgbouncer(6432)psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta # pg-meta-default : Direct connect primary via primary postgres(5432)psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # pg-meta-offline : Direct connect offline via offline postgres(5432)
Here the pg-meta domain name is point to the cluster’s L2 VIP, which in turn points to the haproxy load balancer on the primary instance.
It is responsible for routing traffic to different instances, check Access Services for details.
Primary Service
The primary service may be the most critical service for production usage.
It means all cluster members will be included in the primary service (selector: "[]"), but the one and only one instance that past health check (check: /primary) will be used as the primary instance. Patroni will guarantee that only one instance is primary at any time, so the primary service will always route traffic to THE primary instance.
listen pg-test-primarybind *:5433mode tcpmaxconn 5000balance roundrobinoption httpchkoption http-keep-alivehttp-check send meth OPTIONS uri /primaryhttp-check expect status 200default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100# serversserver pg-test-1 10.10.10.11:6432 check port 8008 weight 100server pg-test-3 10.10.10.13:6432 check port 8008 weight 100server pg-test-2 10.10.10.12:6432 check port 8008 weight 100
Replica Service
The replica service is used for production read-only traffic.
There may be many more read-only queries than read-write queries in real-world scenarios. You may have many replicas.
The replica service traffic will try to use common pg instances with pg_role = replica to alleviate the load on the primary instance as much as possible. It will try NOT to use instances with pg_role = offline to avoid mixing OLAP & OLTP queries as much as possible.
All cluster members will be included in the replica service (selector: "[]") when it passes the read-only health check (check: /read-only). primary and offline instances are used as backup servers, which will take over in case of all replica instances are down.
listen pg-test-replicabind *:5434mode tcpmaxconn 5000balance roundrobinoption httpchkoption http-keep-alivehttp-check send meth OPTIONS uri /read-onlyhttp-check expect status 200default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100# serversserver pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backupserver pg-test-3 10.10.10.13:6432 check port 8008 weight 100server pg-test-2 10.10.10.12:6432 check port 8008 weight 100
Default Service
The default service will route to primary postgres (5432) by default.
It is quite like the primary service, except it will always bypass pgbouncer, regardless of pg_default_service_dest. Which is useful for administration connection, ETL writes, CDC changing data capture, etc…
listen pg-test-offlinebind *:5438mode tcpmaxconn 5000balance roundrobinoption httpchkoption http-keep-alivehttp-check send meth OPTIONS uri /replicahttp-check expect status 200default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100# serversserver pg-test-3 10.10.10.13:5432 check port 8008 weight 100server pg-test-2 10.10.10.12:5432 check port 8008 weight 100 backup
Access Service
Pigsty exposes service with haproxy. Which is enabled on all nodes by default.
haproxy load balancers are idempotent among the same pg cluster by default, you use ANY / ALL of them by all means.
The typical method is access via cluster domain name, which resolves to cluster L2 VIP, or all instances ip address in a round-robin manner.
Service can be implemented in different ways. You can even implement your own access method such as L4 LVS, F5, etc… instead of haproxy.
You can use a different combination of host & port, they are providing PostgreSQL service in different ways.
Host
type
sample
description
Cluster Domain Name
pg-test
via cluster domain name (resolved by dnsmasq @ infra nodes)
Cluster VIP Address
10.10.10.3
via a L2 VIP address managed by vip-manager, bind to primary
Instance Hostname
pg-test-1
Access via any instance hostname (resolved by dnsmasq @ infra nodes)
Instance IP Address
10.10.10.11
Access any instance ip address
Port
Pigsty uses different ports to distinguish between pg services:
port
service
type
description
5432
postgres
database
Direct access to postgres server
6432
pgbouncer
middleware
Go through connection pool middleware before postgres
5433
primary
service
Access primary pgbouncer (or postgres)
5434
replica
service
Access replica pgbouncer (or postgres)
5436
default
service
Access primary postgres
5438
offline
service
Access offline postgres
Combinations
# Access via cluster domainpostgres://test@pg-test:5432/test # DNS -> L2 VIP -> primary direct connectionpostgres://test@pg-test:6432/test # DNS -> L2 VIP -> primary connection pool -> primarypostgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary Connection Pool -> Primarypostgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica Connection Pool -> Replicapostgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for Admin)postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> offline direct connection (for ETL/personal queries)# Direct access via cluster VIPpostgres://[email protected]:5432/test # L2 VIP -> Primary direct accesspostgres://[email protected]:6432/test # L2 VIP -> Primary Connection Pool -> Primarypostgres://[email protected]:5433/test # L2 VIP -> HAProxy -> Primary Connection Pool -> Primarypostgres://[email protected]:5434/test # L2 VIP -> HAProxy -> Repilca Connection Pool -> Replicapostgres://[email protected]:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for Admin)postgres://[email protected]::5438/test # L2 VIP -> HAProxy -> offline direct connect (for ETL/personal queries)# Specify any cluster instance name directlypostgres://test@pg-test-1:5432/test # DNS -> Database Instance Direct Connect (singleton access)postgres://test@pg-test-1:6432/test # DNS -> connection pool -> databasepostgres://test@pg-test-1:5433/test # DNS -> HAProxy -> connection pool -> database read/writepostgres://test@pg-test-1:5434/test # DNS -> HAProxy -> connection pool -> database read-onlypostgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> database direct connectpostgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> database offline read/write# Directly specify any cluster instance IP accesspostgres://[email protected]:5432/test # Database instance direct connection (directly specify instance, no automatic traffic distribution)postgres://[email protected]:6432/test # Connection Pool -> Databasepostgres://[email protected]:5433/test # HAProxy -> connection pool -> database read/writepostgres://[email protected]:5434/test # HAProxy -> connection pool -> database read-onlypostgres://[email protected]:5436/test # HAProxy -> Database Direct Connectionspostgres://[email protected]:5438/test # HAProxy -> database offline read-write# Smart client automatic read/write separation (connection pooling)postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby
11.11 - Auth / HBA
Host-Based Authentication in Pigsty
PostgreSQL has various authentication methods. You can use all of them, while pigsty’s battery-included ACL system focuses on HBA, password, and SSL authentication.
Client Authentication
To connect to a PostgreSQL database, the user has to be authenticated (with a password by default).
You can provide the password in the connection string (not secure) or use the PGPASSWORD env or .pgpass file. Check psql docs and PostgreSQL connection string for more details.
Typically, global HBA is defined in all.vars. If you want to modify the global default HBA rules, you can copy from the full.yml template to all.vars for modification.
Here are some examples of cluster HBA rule definitions.
pg-meta:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-metapg_hba_rules:- {user:dbuser_view ,db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}- {user:all ,db:all ,addr:100.0.0.0/8 ,auth:pwd ,title:'all user access all db from kubernetes cluster'}- {user:'${admin}',db:world ,addr:0.0.0.0/0 ,auth:cert ,title:'all admin world access with client cert'}
Reload HBA
To reload postgres/pgbouncer hba rules:
bin/pgsql-hba <cls> # reload hba rules of cluster `<cls>`bin/pgsql-hba <cls> ip1 ip2... # reload hba rules of specific instances
Pigsty has a default set of HBA rules, which is pretty secure for most cases.
The rules are self-explained in alias form.
pg_default_hba_rules:# postgres default host-based authentication rules- {user:'${dbsu}',db:all ,addr:local ,auth:ident ,title:'dbsu access via local os user ident'}- {user:'${dbsu}',db:replication ,addr:local ,auth:ident ,title:'dbsu replication from local os ident'}- {user:'${repl}',db:replication ,addr:localhost ,auth:pwd ,title:'replicator replication from localhost'}- {user:'${repl}',db:replication ,addr:intra ,auth:pwd ,title:'replicator replication from intranet'}- {user:'${repl}',db:postgres ,addr:intra ,auth:pwd ,title:'replicator postgres db from intranet'}- {user:'${monitor}',db:all ,addr:localhost ,auth:pwd ,title:'monitor from localhost with password'}- {user:'${monitor}',db:all ,addr:infra ,auth:pwd ,title:'monitor from infra host with password'}- {user:'${admin}',db:all ,addr:infra ,auth:ssl ,title:'admin @ infra nodes with pwd & ssl'}- {user:'${admin}',db:all ,addr:world ,auth:ssl ,title:'admin @ everywhere with ssl & pwd'}- {user:'+dbrole_readonly',db:all ,addr:localhost ,auth:pwd ,title:'pgbouncer read/write via local socket'}- {user:'+dbrole_readonly',db:all ,addr:intra ,auth:pwd ,title:'read/write biz user via password'}- {user:'+dbrole_offline' ,db:all ,addr:intra ,auth:pwd ,title:'allow etl offline tasks from intranet'}pgb_default_hba_rules:# pgbouncer default host-based authentication rules- {user:'${dbsu}',db:pgbouncer ,addr:local ,auth:peer ,title:'dbsu local admin access with os ident'}- {user:'all' ,db:all ,addr:localhost ,auth:pwd ,title:'allow all user local access with pwd'}- {user:'${monitor}',db:pgbouncer ,addr:intra ,auth:pwd ,title:'monitor access via intranet with pwd'}- {user:'${monitor}',db:all ,addr:world ,auth:deny ,title:'reject all other monitor access addr'}- {user:'${admin}',db:all ,addr:intra ,auth:pwd ,title:'admin access via intranet with pwd'}- {user:'${admin}',db:all ,addr:world ,auth:deny ,title:'reject all other admin access addr'}- {user:'all' ,db:all ,addr:intra ,auth:pwd ,title:'allow all user intra access with pwd'}
Security Enhancement
For those critical cases, we have a safe.yml template with the following hba rule set as a reference:
pg_default_hba_rules:# postgres host-based auth rules by default- {user:'${dbsu}',db:all ,addr:local ,auth:ident ,title:'dbsu access via local os user ident'}- {user:'${dbsu}',db:replication ,addr:local ,auth:ident ,title:'dbsu replication from local os ident'}- {user:'${repl}',db:replication ,addr:localhost ,auth:ssl ,title:'replicator replication from localhost'}- {user:'${repl}',db:replication ,addr:intra ,auth:ssl ,title:'replicator replication from intranet'}- {user:'${repl}',db:postgres ,addr:intra ,auth:ssl ,title:'replicator postgres db from intranet'}- {user:'${monitor}',db:all ,addr:localhost ,auth:pwd ,title:'monitor from localhost with password'}- {user:'${monitor}',db:all ,addr:infra ,auth:ssl ,title:'monitor from infra host with password'}- {user:'${admin}',db:all ,addr:infra ,auth:ssl ,title:'admin @ infra nodes with pwd & ssl'}- {user:'${admin}',db:all ,addr:world ,auth:cert ,title:'admin @ everywhere with ssl & cert'}- {user:'+dbrole_readonly',db:all ,addr:localhost ,auth:ssl ,title:'pgbouncer read/write via local socket'}- {user:'+dbrole_readonly',db:all ,addr:intra ,auth:ssl ,title:'read/write biz user via password'}- {user:'+dbrole_offline' ,db:all ,addr:intra ,auth:ssl ,title:'allow etl offline tasks from intranet'}pgb_default_hba_rules:# pgbouncer host-based authentication rules- {user:'${dbsu}',db:pgbouncer ,addr:local ,auth:peer ,title:'dbsu local admin access with os ident'}- {user:'all' ,db:all ,addr:localhost ,auth:pwd ,title:'allow all user local access with pwd'}- {user:'${monitor}',db:pgbouncer ,addr:intra ,auth:ssl ,title:'monitor access via intranet with pwd'}- {user:'${monitor}',db:all ,addr:world ,auth:deny ,title:'reject all other monitor access addr'}- {user:'${admin}',db:all ,addr:intra ,auth:ssl ,title:'admin access via intranet with pwd'}- {user:'${admin}',db:all ,addr:world ,auth:deny ,title:'reject all other admin access addr'}- {user:'all' ,db:all ,addr:intra ,auth:ssl ,title:'allow all user intra access with pwd'}
!> Remember to change these password in production deployment !
pg_dbsu:postgres # os user for the databasepg_replication_username:replicator # system replication userpg_replication_password:DBUser.Replicator # system replication passwordpg_monitor_username:dbuser_monitor # system monitor userpg_monitor_password:DBUser.Monitor # system monitor passwordpg_admin_username:dbuser_dba # system admin userpg_admin_password:DBUser.DBA # system admin password
- GRANT USAGE ON SCHEMAS TO dbrole_readonly- GRANT SELECT ON TABLES TO dbrole_readonly- GRANT SELECT ON SEQUENCES TO dbrole_readonly- GRANT EXECUTE ON FUNCTIONS TO dbrole_readonly- GRANT USAGE ON SCHEMAS TO dbrole_offline- GRANT SELECT ON TABLES TO dbrole_offline- GRANT SELECT ON SEQUENCES TO dbrole_offline- GRANT EXECUTE ON FUNCTIONS TO dbrole_offline- GRANT INSERT ON TABLES TO dbrole_readwrite- GRANT UPDATE ON TABLES TO dbrole_readwrite- GRANT DELETE ON TABLES TO dbrole_readwrite- GRANT USAGE ON SEQUENCES TO dbrole_readwrite- GRANT UPDATE ON SEQUENCES TO dbrole_readwrite- GRANT TRUNCATE ON TABLES TO dbrole_admin- GRANT REFERENCES ON TABLES TO dbrole_admin- GRANT TRIGGER ON TABLES TO dbrole_admin- GRANT CREATE ON SCHEMAS TO dbrole_admin
Newly created objects will have corresponding privileges when it is created by admin users
The \ddp+ may looks like:
Type
Access privileges
function
=X
dbrole_readonly=X
dbrole_offline=X
dbrole_admin=X
schema
dbrole_readonly=U
dbrole_offline=U
dbrole_admin=UC
sequence
dbrole_readonly=r
dbrole_offline=r
dbrole_readwrite=wU
dbrole_admin=rwU
table
dbrole_readonly=r
dbrole_offline=r
dbrole_readwrite=awd
dbrole_admin=arwdDxt
Default Privilege
ALTER DEFAULT PRIVILEGES allows you to set the privileges that will be applied to objects created in the future. It does not affect privileges assigned to already-existing objects, and objects created by non-admin users.
Pigsty will use the following default privileges:
{%forprivinpg_default_privileges%}ALTERDEFAULTPRIVILEGESFORROLE{{pg_dbsu}}{{priv}};{%endfor%}{%forprivinpg_default_privileges%}ALTERDEFAULTPRIVILEGESFORROLE{{pg_admin_username}}{{priv}};{%endfor%}-- for additional business admin, they can SET ROLE to dbrole_admin
{%forprivinpg_default_privileges%}ALTERDEFAULTPRIVILEGESFORROLE"dbrole_admin"{{priv}};{%endfor%}
Which will be rendered in pg-init-template.sql alone with ALTER DEFAULT PRIVILEGES statement for admin users.
These SQL commands will be executed on postgres & template1 during cluster bootstrap, and newly created databases will inherit it from template1 by default.
That is to say, to maintain the correct object privilege, you have to run DDL with admin users, which could be:
It’s wise to use postgres as the global object owner to perform DDL changes.
If you wish to create objects with business admin user, YOU MUST USE SET ROLE dbrole_admin before running that DDL to maintain the correct privileges.
You can also ALTER DEFAULT PRIVILEGE FOR ROLE <some_biz_admin> XXX to grant default privilege to business admin user, too.
There are 3 database level privileges: CONNECT, CREATE, TEMP, and a special ‘privilege’: OWNERSHIP.
- name:meta # required, `name` is the only mandatory field of a database definitionowner:postgres # optional, specify a database owner, {{ pg_dbsu }} by defaultallowconn:true# optional, allow connection, true by default. false will disable connect at allrevokeconn:false# optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
If owner exists, it will be used as the database owner instead of default {{ pg_dbsu }}
If revokeconn is false, all users have the CONNECT privilege of the database, this is the default behavior.
If revokeconn is set to true explicitly:
CONNECT privilege of the database will be revoked from PUBLIC
CONNECT privilege will be granted to {{ pg_replication_username }}, {{ pg_monitor_username }} and {{ pg_admin_username }}
CONNECT privilege will be granted to the database owner with GRANT OPTION
revokeconn flag can be used for database access isolation, you can create different business users as the owners for each database and set the revokeconn option for all of them.
Create Privilege
Pigsty revokes the CREATE privilege on database from PUBLIC by default, for security consideration. And this is the default behavior since PostgreSQL 15.
The database owner has the full ability to adjust these privileges as they see fit.
11.13 - Dashboard
check visualized information
Grafana Dashboards for PostgreSQL clusters: Demo & Gallery.
There are 26 default grafana dashboards about PostgreSQL and categorized into 4 levels. and categorized into PGSQL, PGCAT & PGLOG by datasource.
PGSQL Database: The main dashboard for a single PGSQL database
PGSQL Tables : Table/Index access metrics inside a single database
PGSQL Table: Detailed information (QPS/RT/Index/Seq…) about a single table
PGSQL Query: Detailed information (QPS/RT) about a single query
PGCAT
PGCAT Instance: Instance information from database catalog directly
PGCAT Database: Database information from database catalog directly
PGCAT Schema: Detailed information about one single schema from database catalog directly
PGCAT Table: Detailed information about one single table from database catalog directly
PGCAT Query: Detailed information about one single type of query from database catalog directly
PGCAT Locks: Detailed information about live locks & activity from database catalog directly
PGLOG
PGLOG Overview: Overview of csv log sample in pigsty meta database
PGLOG Overview: Detail of one single session of csv log sample in pigsty meta database
11.14 - Migration
zero-downtime blue-green deployment
Pigsty has a built-in playbook pgsql-migration.yml to perform online database migration based on logical replication.
With proper automation, the downtime could be minimized to several seconds. But beware that logical replication requires PostgreSQL 10+ to work.
You can still use the facility here and use a pg_dump | psql instead of logical replication.
Define Migration Task
You have to create a migration task definition file to use this playbook.
You have to tell pigsty where is the source cluster and destination cluster. The database to be migrated, and the primary IP address.
You should have superuser privileges on both sides to proceed
You can overwrite the superuser connection to the source cluster with src_pg, and logical replication connection string with sub_conn, Otherwise, pigsty default admin & replicator credentials will be used.
---#-----------------------------------------------------------------# PG_MIGRATION#-----------------------------------------------------------------context_dir:~/migration # migration manuals & scripts#-----------------------------------------------------------------# SRC Cluster (The OLD Cluster)#-----------------------------------------------------------------src_cls:pg-meta # src cluster name <REQUIRED>src_db:meta # src database name <REQUIRED>src_ip:10.10.10.10# src cluster primary ip <REQUIRED>#src_pg: '' # if defined, use this as src dbsu pgurl instead of:# # postgres://{{ pg_admin_username }}@{{ src_ip }}/{{ src_db }}# # e.g. 'postgres://dbuser_dba:[email protected]:5432/meta'#sub_conn: '' # if defined, use this as subscription connstr instead of:# # host={{ src_ip }} dbname={{ src_db }} user={{ pg_replication_username }}'# # e.g. 'host=10.10.10.10 dbname=meta user=replicator password=DBUser.Replicator'#-----------------------------------------------------------------# DST Cluster (The New Cluster)#-----------------------------------------------------------------dst_cls:pg-test # dst cluster name <REQUIRED>dst_db:test # dst database name <REQUIRED>dst_ip:10.10.10.11# dst cluster primary ip <REQUIRED>#dst_pg: '' # if defined, use this as dst dbsu pgurl instead of:# # postgres://{{ pg_admin_username }}@{{ dst_ip }}/{{ dst_db }}# # e.g. 'postgres://dbuser_dba:[email protected]:5432/test'#-----------------------------------------------------------------# PGSQL#-----------------------------------------------------------------pg_dbsu:postgrespg_replication_username:replicatorpg_replication_password:DBUser.Replicatorpg_admin_username:dbuser_dbapg_admin_password:DBUser.DBApg_monitor_username:dbuser_monitorpg_monitor_password:DBUser.Monitor#-----------------------------------------------------------------...
Generate Plan
The playbook does not migrate src to dst, but it will generate everything your need to do so.
After the execution, you will find migration context dir under ~/migration/pg-meta.meta by default
Following the README.md and executing these scripts one by one, you will do the trick!
# this script will setup migration context with env vars. ~/migration/pg-meta.meta/activate
# these scripts are used for check src cluster status# and help generating new cluster definition in pigsty./check-user # check src users./check-db # check src databases./check-hba # check src hba rules./check-repl # check src replica identities./check-misc # check src special objects# these scripts are used for building logical replication# between existing src cluster and pigsty managed dst cluster# schema, data will be synced in realtime, except for sequences./copy-schema # copy schema to dest./create-pub # create publication on src./create-sub # create subscription on dst./copy-progress # print logical replication progress./copy-diff # quick src & dst diff by counting tables# these scripts will run in an online migration, which will# stop src cluster, copy sequence numbers (which is not synced with logical replication)# you have to reroute you app traffic according to your access method (dns,vip,haproxy,pgbouncer,etc...)# then perform cleanup to drop subscription and publication./copy-seq [n]# sync sequence numbers, if n is given, an additional shift will applied#./disable-src # restrict src cluster access to admin node & new cluster (YOUR IMPLEMENTATION)#./re-routing # ROUTING APPLICATION TRAFFIC FROM SRC TO DST! (YOUR IMPLEMENTATION)./drop-sub # drop subscription on dst after migration./drop-pub # drop publication on src after migration
Caveats
You can use ./copy-seq 1000 to advance all sequences by a number (e.g. 1000) after syncing sequences. Which may prevent potential serial primary key conflict in new clusters.
You have to implement your own ./re-routing script to route your application traffic from src to dst. Since we don’t know how your traffic is routed (e.g dns, VIP, haproxy, or pgbouncer). Of course, you can always do that by hand…
You have to implement your own ./disable-src script to restrict the src cluster. You can do that by changing HBA rules & reload (recommended), or just shutting down postgres, pgbouncer, or haproxy…
11.15 - Backup
Backup and point-in-time recovery
Pigsty uses pgBackRest to manage PostgreSQL backups, it may be the most powerful open-source backup tools in the ecosystem.
With incremental / parallel backup & restore, encryption, MinIO / S3 support, and many other features.
Pigsty has pre-configured it for every PGSQL cluster by default.
Policy
Backup scripts, scheduling, pgbackrest, repo and admin
Admin
Backup policy, disk planning, recovery window trade-off
Restore
Restore to specific time point with playbook
Example
Sandbox example: Perform recovery with bare hands
NO WRANTTY
Pigsty try its best to provide a reliable PITR solution, but we do not take any responsibility for the data loss caused by the PITR operation, use it at your own risk.
For professional support, consider our pro service.
Quick Start
Step 1
[Backup Policy](/docs/pgsql/backup/mechanism): Schedule Base Backups with Crontab
Step 2
[WAL Archiving](/docs/pgsql/backup/policy): Continuously record write activities
Step 3
[Restore & Recovery](/docs/pgsql/backup/restore): Recover from backup and wal archive
Backup script, schedule, repository, and infrastructure
Backups can be invoked by built-in scripts, scheduled with node crontab,
managed by pgbackrest, and stored in backup repo,
which could be local disk filesystem or MinIO / S3, with different retention policies.
Script
You can create a backup with pgbackrest command with pg_dbsu user (postgres by default):
pgbackrest --stanza=pg-meta --type=full backup # create a full backup for cluster pg-meta
The stanza here is the database cluster name: pg_cluster, which is pg-meta for the default setup.
Pigsty has an alias pb and wrapper script pg-backup that fills the current cluster name as stanza:
alias
function pb(){localstanza=$(grep -o '\[[^][]*]' /etc/pgbackrest/pgbackrest.conf | head -n1 | sed 's/.*\[\([^]]*\)].*/\1/') pgbackrest --stanza=$stanza$@}pb ... # pgbackrest --stanza=pg-meta ...pb info # pgbackrest --stanza=pg-meta infopb backup # pgbackrest --stanza=pg-meta backup
script
pg-backup full # take an full backup = pgbackrest --stanza=pg-meta --type=incr backuppg-backup incr # take an incremental backup = pgbackrest --stanza=pg-meta --type=incr backuppg-backup diff # take an differential backup = pgbackrest --stanza=pg-meta --type=incr backup
Crontab
Pigsty is leveraging Linux’s crontab to schedule backups. You can define your backup policies with it
For example, most one-node config template will have the following node_crontab for backup.
You can design more sophisticated backup policies with crontab and pg-backup script, such as:
Full backup on Monday, incremental backup during weekdays
node_crontab:# make a full backup on monday 1am, and an incremental backup during weekdays- '00 01 * * 1 postgres /pg/bin/pg-backup full'- '00 01 * * 2,3,4,5,6,7 postgres /pg/bin/pg-backup'
To apply crontab change, use the node.yml to update the crontab on all nodes.
apply crontab
./node.yml -t node_crontab -l pg-meta # apply crontab change to the pg-meta group
pgbackrest
Here’s pigsty’s setup details for pgbackrest:
The pgbackrest backup tool is enabled and configured by default (pgbackrest_enabled)
Installed in the pg_install task in the pgsql.yml playbook, defined in pg_packages
tmp: /pg/spool is used as the temp spool directory for pgbackrest
data: /pg/backup is used, if the default local filesystem backup repo is selected.
Moreover, during the PITR Recovery process,
Pigsty will create a temp /pg/conf/pitr.conf pgbackrest config file.
And write postgres recovery log to the /pg/tmp/recovery.log file.
When a postgres cluster is created, pigsty will create an initial backup automatically.
It’s a tiny backup since the new cluster is almost empty.
It will leave a marker file /etc/pgbackrest/initial.done to avoid creating the initial backup again.
Set the pgbackrest_init_backup to false if you don’t want it.
Administration
Enable Backup
If you database cluster is created with pgbackrest_enable set to true, the backup will be enabled automatically.
If it created with the false value, you can enable the pgbackrest component with:
./pgsql.yml -t pg_backup # run the pgbackrest subtask
Remove Backup
Pigsty will remove pgbackrest backup stanza when removing the primary instance (pg_role = primary).
Use the pg_backup subtask to remove the backup only, and use the pg_rm_backup arg to keep backups.
If your backup repo is locked, (e.g., S3 / MinIO has a lock option), this operation will fail.
Backup Removal
Removing backup may lead to permanent data loss, it’s a dangerous operation, do with extreme caution.
List Backup
This command will list all backups in the pgbackrest repository (shared by all clusters)
pgbackrest info
Manual Backup
Pigsty has a built-in script /pg/bin/pg-backup which wraps the pgbackrest backup command.
pg-backup # take an incremental backuppg-backup full # take an full backuppg-backup incr # take an incremental backuppg-backup diff # take an differential backup
Base Backup
Pigsty has an alternative backup script /pg/bin/pg-basebackup which does not rely on pgbackrest, and gives you a physical copy of the database cluster.
The default backup dir is /pg/backup.
NAME
pg-basebackup -- make base backup from PostgreSQL instance
SYNOPSIS
pg-basebackup -sdfeukr
pg-basebackup --src postgres:/// --dst . --file backup.tar.lz4
DESCRIPTION
-s, --src, --url Backup source URL, optional, "postgres:///" by default, if password is required, it should be given in url, ENV or .pgpass
-d, --dst, --dir Where to put backup files, "/pg/backup" by default
-f, --file Overwrite default backup filename, "backup_${tag}_${date}.tar.lz4"-r, --remove .lz4 Files mtime before n minutes ago will be removed, default is 1200(20hour)-t, --tag Backup file tag, if not set, target cluster_name or local ip address will be used. Also used as part of DEFAULT filename
-k, --key Encryption key when --encrypt is specified, default key is ${tag}-u, --upload Upload backup files to cloud storage, (need your own implementation)-e, --encryption Encrypt with RC4 using OpenSSL, if not key is specified, tag is used as key
-h, --help Print this message
postgres@pg-meta-1:~$ pg-basebackup
[2025-07-13 06:16:05][INFO]================================================================[2025-07-13 06:16:05][INFO][INIT] pg-basebackup begin, checking parameters
[2025-07-13 06:16:05][DEBUG][INIT]#====== BINARY[2025-07-13 06:16:05][DEBUG][INIT] pg_basebackup : /usr/pgsql/bin/pg_basebackup
[2025-07-13 06:16:05][DEBUG][INIT] openssl : /usr/bin/openssl
[2025-07-13 06:16:05][DEBUG][INIT]#====== PARAMETER[2025-07-13 06:16:05][DEBUG][INIT] filename (-f) : backup_pg-meta_20250713.tar.lz4
[2025-07-13 06:16:05][DEBUG][INIT] src (-s) : postgres:///
[2025-07-13 06:16:05][DEBUG][INIT] dst (-d) : /pg/backup
[2025-07-13 06:16:05][DEBUG][INIT] tag (-t) : pg-meta
[2025-07-13 06:16:05][DEBUG][INIT] key (-k) : pg-meta
[2025-07-13 06:16:05][DEBUG][INIT] encrypt (-e) : false[2025-07-13 06:16:05][DEBUG][INIT] upload (-u) : false[2025-07-13 06:16:05][DEBUG][INIT] remove (-r) : -mmin +1200
[2025-07-13 06:16:05][INFO][LOCK] acquire lock @ /tmp/backup.lock
[2025-07-13 06:16:05][INFO][LOCK] lock acquired success on /tmp/backup.lock, pid=107417[2025-07-13 06:16:05][INFO][BKUP] backup begin, from postgres:/// to /pg/backup/backup_pg-meta_20250713.tar.lz4
[2025-07-13 06:16:05][INFO][BKUP] backup in normal mode
pg_basebackup: initiating base backup, waiting for checkpoint to completepg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/7000028 on timeline 1pg_basebackup: write-ahead log end point: 0/7000FD8
pg_basebackup: syncing data to disk ...
pg_basebackup: base backup completed
[2025-07-13 06:16:06][INFO][BKUP] backup complete!
[2025-07-13 06:16:06][INFO][RMBK] remove local obsolete backup: 1200[2025-07-13 06:16:06][INFO][BKUP] find obsolete backups: find /pg/backup/ -maxdepth 1 -type f -mmin +1200 -name 'backup*.lz4'[2025-07-13 06:16:06][WARN][BKUP] remove obsolete backups:
[2025-07-13 06:16:06][INFO][RMBK] remove old backup complete[2025-07-13 06:16:06][INFO][LOCK] release lock @ /tmp/backup.lock
[2025-07-13 06:16:06][INFO][DONE] backup procedure complete!
[2025-07-13 06:16:06][INFO]================================================================
Backup are compressed with lz4, You can unzip and extract the tarball with the following command:
mkdir -p /tmp/data # extract backup to this directorycat /pg/backup/backup_pg-meta_20250713.tar.lz4 | unlz4 -d -c | tar -xC /tmp/data
Logical Backup
You can also use the pg_dump command to perform a logical backup.
Logical backups cannot be used for PITR (Point In Time Recovery),
but they are useful for migrating data between different major versions, or implement flexible data export logic.
Bootstrap from Repo
Now let’s say you have an existing cluster pg-meta, and want to FORK it as pg-meta2:
You’ll need to create the new pg-meta2 cluster fork, then run pitr on it.
11.15.2 - Repository
Backup storage repository for PostgreSQL
You can to configure WHERE to store the backups by specifying the pgbackrest_repo parameter.
You can define multiple repo there, and Pigsty will pick it according to the value of pgbackrest_method.
Default Repo
By default, Pigsty has two default backup repo definition: the local and minio backup repo.
local: The default, use the local /pg/backup dir (Softlink point to pg_fs_backup: /data/backups)
minio: Use the SNSD 1-node MinIO cluster (Supported by pigsty, but not enabled by default)
pgbackrest_method:local # choose the backup repo method, `local` or `minio` or any other user defined repopgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositorylocal:# default pgbackrest repo with local posix fspath:/pg/backup # local backup directory, `/pg/backup` by defaultretention_full_type:count # retention full backups by countretention_full:2# keep 2, at most 3 full backups when using local fs repominio:# optional minio repo for pgbackresttype:s3 # minio is s3-compatible, so s3 is useds3_endpoint:sss.pigsty # minio endpoint domain name, `sss.pigsty` by defaults3_region:us-east-1 # minio region, us-east-1 by default, useless for minios3_bucket:pgsql # minio bucket name, `pgsql` by defaults3_key:pgbackrest # minio user access key for pgbackrests3_key_secret:S3User.Backup # minio user secret key for pgbackrests3_uri_style:path # use path style uri for minio rather than host stylepath:/pgbackrest # minio backup path, default is `/pgbackrest`storage_port:9000# minio port, 9000 by defaultstorage_ca_file:/etc/pki/ca.crt # minio ca file path, `/etc/pki/ca.crt` by defaultblock:y# Enable block incremental backupbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest # AES encryption password, default is 'pgBackRest'retention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for the last 14 days
Repo Retention
If you take backups every day without deleting them, the backup repo will grow larger and larger and blow your disk space.
You’ll need to define a retention policy to only keep a limited number of backups.
The default backup policy is defined in the pgbackrest_repo parameter, change them on demand.
local: keep last 2 full backups, at most 3 during backup
minio: keep all full backups in the last 14 days
Space Planning
Object storage provides virtually unlimited storage capacity, so you don’t need to worry about the disk space.
You can optimize space usage with a hybrid full & diff backup policy.
For local disk backup repo, pigsty recommends using a retention policy of keeping the last 2 full backups,
which means keep the two most-recent full backups on disk (a third copy may exist while a new backup is running).
This gives you a guaranteed recovery window of at least last 24 hours. Check backup policy for details.
Object storage service provides virtually unlimited storage capacity, and provides a remote disaster tolerance for your system.
If you don’t have one, Pigsty has built-in MinIO support.
MinIO
You can enable minio backup repo by uncommenting the following settings.
Beware that pgbackrest only takes HTTPS / domain names, so you have to run MinIO with a domain name and HTTPS endpoint.
all:vars:pgbackrest_method:minio # use minio as the default backup repochildren:# define a one-node minio SNSD clusterminio:{hosts:{10.10.10.10:{minio_seq:1 }} ,vars:{minio_cluster:minio }}
S3
If you only have one node, the meaningful backup policy could be using a cloud vendor’s object storage service such as AWS S3, Aliyun OSS, or Google Cloud, etc…
To achieve this, you can define a new repo:
pgbackrest_method:s3 # use the 'pgbackrest_repo.s3' as backup repopgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositorys3:# aliyun oss (s3 compatible) object storage servicetype:s3 # oss is s3-compatibles3_endpoint:oss-cn-beijing-internal.aliyuncs.coms3_region:oss-cn-beijings3_bucket:<your_bucket_name>s3_key:<your_access_key>s3_key_secret:<your_secret_key>s3_uri_style:hostpath:/pgbackrestbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest # AES encryption password, default is 'pgBackRest'retention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for last 14 dayslocal:# default pgbackrest repo with local posix fspath:/pg/backup # local backup directory, `/pg/backup` by defaultretention_full_type:count # retention full backups by countretention_full:2# keep 2, at most 3 full backups when using local fs repo
Manage Backups
Enable Backup
If you database cluster is created with pgbackrest_enable set to true, the backup will be enabled automatically.
If it created with the false value, you can enable the pgbackrest component with:
./pgsql.yml -t pg_backup # run the pgbackrest subtask
Remove Backup
Pigsty will remove pgbackrest backup stanza when removing the primary instance (pg_role = primary).
Use the pg_backup subtask to remove the backup only, and use the pg_rm_backup arg to keep backups.
If your backup repo is locked, (e.g., S3 / MinIO has a lock option), this operation will fail.
Backup Removal
Removing backup may lead to permanent data loss, it’s a dangerous operation, do with extreme caution.
List Backup
This command will list all backups in the pgbackrest repository (shared by all clusters)
pgbackrest info
Manual Backup
Pigsty has a built-in script /pg/bin/pg-backup which wraps the pgbackrest backup command.
pg-backup # take an incremental backuppg-backup full # take an full backuppg-backup incr # take an incremental backuppg-backup diff # take an differential backup
Base Backup
Pigsty has an alternative backup script /pg/bin/pg-basebackup which does not rely on pgbackrest, and gives you a physical copy of the database cluster.
The default backup dir is /pg/backup.
NAME
pg-basebackup -- make base backup from PostgreSQL instance
SYNOPSIS
pg-basebackup -sdfeukr
pg-basebackup --src postgres:/// --dst . --file backup.tar.lz4
DESCRIPTION
-s, --src, --url Backup source URL, optional, "postgres:///" by default, if password is required, it should be given in url, ENV or .pgpass
-d, --dst, --dir Where to put backup files, "/pg/backup" by default
-f, --file Overwrite default backup filename, "backup_${tag}_${date}.tar.lz4"-r, --remove .lz4 Files mtime before n minutes ago will be removed, default is 1200(20hour)-t, --tag Backup file tag, if not set, target cluster_name or local ip address will be used. Also used as part of DEFAULT filename
-k, --key Encryption key when --encrypt is specified, default key is ${tag}-u, --upload Upload backup files to cloud storage, (need your own implementation)-e, --encryption Encrypt with RC4 using OpenSSL, if not key is specified, tag is used as key
-h, --help Print this message
postgres@pg-meta-1:~$ pg-basebackup
[2025-07-13 06:16:05][INFO]================================================================[2025-07-13 06:16:05][INFO][INIT] pg-basebackup begin, checking parameters
[2025-07-13 06:16:05][DEBUG][INIT]#====== BINARY[2025-07-13 06:16:05][DEBUG][INIT] pg_basebackup : /usr/pgsql/bin/pg_basebackup
[2025-07-13 06:16:05][DEBUG][INIT] openssl : /usr/bin/openssl
[2025-07-13 06:16:05][DEBUG][INIT]#====== PARAMETER[2025-07-13 06:16:05][DEBUG][INIT] filename (-f) : backup_pg-meta_20250713.tar.lz4
[2025-07-13 06:16:05][DEBUG][INIT] src (-s) : postgres:///
[2025-07-13 06:16:05][DEBUG][INIT] dst (-d) : /pg/backup
[2025-07-13 06:16:05][DEBUG][INIT] tag (-t) : pg-meta
[2025-07-13 06:16:05][DEBUG][INIT] key (-k) : pg-meta
[2025-07-13 06:16:05][DEBUG][INIT] encrypt (-e) : false[2025-07-13 06:16:05][DEBUG][INIT] upload (-u) : false[2025-07-13 06:16:05][DEBUG][INIT] remove (-r) : -mmin +1200
[2025-07-13 06:16:05][INFO][LOCK] acquire lock @ /tmp/backup.lock
[2025-07-13 06:16:05][INFO][LOCK] lock acquired success on /tmp/backup.lock, pid=107417[2025-07-13 06:16:05][INFO][BKUP] backup begin, from postgres:/// to /pg/backup/backup_pg-meta_20250713.tar.lz4
[2025-07-13 06:16:05][INFO][BKUP] backup in normal mode
pg_basebackup: initiating base backup, waiting for checkpoint to completepg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/7000028 on timeline 1pg_basebackup: write-ahead log end point: 0/7000FD8
pg_basebackup: syncing data to disk ...
pg_basebackup: base backup completed
[2025-07-13 06:16:06][INFO][BKUP] backup complete!
[2025-07-13 06:16:06][INFO][RMBK] remove local obsolete backup: 1200[2025-07-13 06:16:06][INFO][BKUP] find obsolete backups: find /pg/backup/ -maxdepth 1 -type f -mmin +1200 -name 'backup*.lz4'[2025-07-13 06:16:06][WARN][BKUP] remove obsolete backups:
[2025-07-13 06:16:06][INFO][RMBK] remove old backup complete[2025-07-13 06:16:06][INFO][LOCK] release lock @ /tmp/backup.lock
[2025-07-13 06:16:06][INFO][DONE] backup procedure complete!
[2025-07-13 06:16:06][INFO]================================================================
Backup are compressed with lz4, You can unzip and extract the tarball with the following command:
mkdir -p /tmp/data # extract backup to this directorycat /pg/backup/backup_pg-meta_20250713.tar.lz4 | unlz4 -d -c | tar -xC /tmp/data
Logical Backup
You can also use the pg_dump command to perform a logical backup.
Logical backups cannot be used for PITR (Point In Time Recovery),
but they are useful for migrating data between different major versions, or implement flexible data export logic.
Bootstrap from Repo
Now let’s say you have an existing cluster pg-meta, and want to FORK it as pg-meta2:
You’ll need to create the new pg-meta2 cluster fork, then run pitr on it.
11.15.3 - Policy
Design backup policy according to your needs.
WHEN: Backup Policy
WHERE: Backup Repo
HOW: Backup Method
WHEN
The first problem is WHEN to backup your database — Trade off between backup frequency and recovery time.
Since you’ll need to replay the WAL logs to your recovery target since the last previous backup,
the more frequent you backup, the less WAL logs you’ll need to replay, and the faster your recovery will be.
Everyday Full Backup
For a production database, it is recommended to start with the simplest everyday full backup policy.
Where is the default backup policy in pigsty, implemented with crontab.
Full backup everyday 1am
node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']pgbackrest_method:local # choose the backup repo method, `local` or `minio` or any other user defined repopgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositorylocal:# default pgbackrest repo with local posix fspath:/pg/backup # local backup directory, `/pg/backup` by defaultretention_full_type:count # retention full backups by countretention_full:2# keep 2, at most 3 full backups when using local fs repo
When using with the default local filesystem backup repo, it provides a 24~48h recovery window.
Let’s assume your database size is 100GB, and 10GB writes per day, and your backup size will be.
It will consume 2 ~ 3x of your database size, plus a 2 day’s WAL.
So in practice, you may have to prepare a backup disk with at least 3 ~ 5x of your database size
to use the default backup policy.
Full + Incr Backup
You can optimize backup space usage by changing these parameters.
If you are using MinIO / S3 as centralized backup repo, you can use more space than your disk limitation.
Then consider the full + incr backup with 2-week retention policy:
node_crontab:# make a full backup on monday 1am, and an incremental backup during weekdays- '00 01 * * 1 postgres /pg/bin/pg-backup full'- '00 01 * * 2,3,4,5,6,7 postgres /pg/bin/pg-backup'pgbackrest_method:miniopgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositoryminio:# optional minio repo for pgbackresttype:s3 # minio is s3-compatible, so s3 is useds3_endpoint:sss.pigsty # minio endpoint domain name, `sss.pigsty` by defaults3_region:us-east-1 # minio region, us-east-1 by default, useless for minios3_bucket:pgsql # minio bucket name, `pgsql` by defaults3_key:pgbackrest # minio user access key for pgbackrests3_key_secret:S3User.Backup # minio user secret key for pgbackrests3_uri_style:path # use path style uri for minio rather than host stylepath:/pgbackrest # minio backup path, default is `/pgbackrest`storage_port:9000# minio port, 9000 by defaultstorage_ca_file:/etc/pki/ca.crt # minio ca file path, `/etc/pki/ca.crt` by defaultblock:y# Enable block incremental backupbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest # AES encryption password, default is 'pgBackRest'retention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for the last 14 days
When using with the built-in minio filesystem backup repo, it provides a guaranteed 1-week pitr window.
Let’s assume your database size is 100GB, and 10GB writes per day, and your backup size will be like:
Where
By default, Pigsty has two default backup repo definition: the local and minio backup repo.
local: The default, use the local /pg/backup dir (Softlink point to pg_fs_backup: /data/backups)
minio: Use the SNSD 1-node MinIO cluster (Supported by pigsty, but not enabled by default)
pgbackrest_method:local # choose the backup repo method, `local` or `minio` or any other user defined repopgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositorylocal:# default pgbackrest repo with local posix fspath:/pg/backup # local backup directory, `/pg/backup` by defaultretention_full_type:count # retention full backups by countretention_full:2# keep 2, at most 3 full backups when using local fs repominio:# optional minio repo for pgbackresttype:s3 # minio is s3-compatible, so s3 is useds3_endpoint:sss.pigsty # minio endpoint domain name, `sss.pigsty` by defaults3_region:us-east-1 # minio region, us-east-1 by default, useless for minios3_bucket:pgsql # minio bucket name, `pgsql` by defaults3_key:pgbackrest # minio user access key for pgbackrests3_key_secret:S3User.Backup # minio user secret key for pgbackrests3_uri_style:path # use path style uri for minio rather than host stylepath:/pgbackrest # minio backup path, default is `/pgbackrest`storage_port:9000# minio port, 9000 by defaultstorage_ca_file:/etc/pki/ca.crt # minio ca file path, `/etc/pki/ca.crt` by defaultblock:y# Enable block incremental backupbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest # AES encryption password, default is 'pgBackRest'retention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for the last 14 days
11.15.4 - Admin
Manage backup repo and backups
Enable Backup
If you database cluster is created with pgbackrest_enable set to true, the backup will be enabled automatically.
If it created with the false value, you can enable the pgbackrest component with:
./pgsql.yml -t pg_backup # run the pgbackrest subtask
Remove Backup
Pigsty will remove pgbackrest backup stanza when removing the primary instance (pg_role = primary).
Use the pg_backup subtask to remove the backup only, and use the pg_rm_backup arg to keep backups.
If your backup repo is locked, (e.g., S3 / MinIO has a lock option), this operation will fail.
Backup Removal
Removing backup may lead to permanent data loss, it’s a dangerous operation, do with extreme caution.
List Backup
This command will list all backups in the pgbackrest repository (shared by all clusters)
pgbackrest info
Manual Backup
Pigsty has a built-in script /pg/bin/pg-backup which wraps the pgbackrest backup command.
pg-backup # take an incremental backuppg-backup full # take an full backuppg-backup incr # take an incremental backuppg-backup diff # take an differential backup
Base Backup
Pigsty has an alternative backup script /pg/bin/pg-basebackup which does not rely on pgbackrest, and gives you a physical copy of the database cluster.
The default backup dir is /pg/backup.
NAME
pg-basebackup -- make base backup from PostgreSQL instance
SYNOPSIS
pg-basebackup -sdfeukr
pg-basebackup --src postgres:/// --dst . --file backup.tar.lz4
DESCRIPTION
-s, --src, --url Backup source URL, optional, "postgres:///" by default, if password is required, it should be given in url, ENV or .pgpass
-d, --dst, --dir Where to put backup files, "/pg/backup" by default
-f, --file Overwrite default backup filename, "backup_${tag}_${date}.tar.lz4"-r, --remove .lz4 Files mtime before n minutes ago will be removed, default is 1200(20hour)-t, --tag Backup file tag, if not set, target cluster_name or local ip address will be used. Also used as part of DEFAULT filename
-k, --key Encryption key when --encrypt is specified, default key is ${tag}-u, --upload Upload backup files to cloud storage, (need your own implementation)-e, --encryption Encrypt with RC4 using OpenSSL, if not key is specified, tag is used as key
-h, --help Print this message
postgres@pg-meta-1:~$ pg-basebackup
[2025-07-13 06:16:05][INFO]================================================================[2025-07-13 06:16:05][INFO][INIT] pg-basebackup begin, checking parameters
[2025-07-13 06:16:05][DEBUG][INIT]#====== BINARY[2025-07-13 06:16:05][DEBUG][INIT] pg_basebackup : /usr/pgsql/bin/pg_basebackup
[2025-07-13 06:16:05][DEBUG][INIT] openssl : /usr/bin/openssl
[2025-07-13 06:16:05][DEBUG][INIT]#====== PARAMETER[2025-07-13 06:16:05][DEBUG][INIT] filename (-f) : backup_pg-meta_20250713.tar.lz4
[2025-07-13 06:16:05][DEBUG][INIT] src (-s) : postgres:///
[2025-07-13 06:16:05][DEBUG][INIT] dst (-d) : /pg/backup
[2025-07-13 06:16:05][DEBUG][INIT] tag (-t) : pg-meta
[2025-07-13 06:16:05][DEBUG][INIT] key (-k) : pg-meta
[2025-07-13 06:16:05][DEBUG][INIT] encrypt (-e) : false[2025-07-13 06:16:05][DEBUG][INIT] upload (-u) : false[2025-07-13 06:16:05][DEBUG][INIT] remove (-r) : -mmin +1200
[2025-07-13 06:16:05][INFO][LOCK] acquire lock @ /tmp/backup.lock
[2025-07-13 06:16:05][INFO][LOCK] lock acquired success on /tmp/backup.lock, pid=107417[2025-07-13 06:16:05][INFO][BKUP] backup begin, from postgres:/// to /pg/backup/backup_pg-meta_20250713.tar.lz4
[2025-07-13 06:16:05][INFO][BKUP] backup in normal mode
pg_basebackup: initiating base backup, waiting for checkpoint to completepg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/7000028 on timeline 1pg_basebackup: write-ahead log end point: 0/7000FD8
pg_basebackup: syncing data to disk ...
pg_basebackup: base backup completed
[2025-07-13 06:16:06][INFO][BKUP] backup complete!
[2025-07-13 06:16:06][INFO][RMBK] remove local obsolete backup: 1200[2025-07-13 06:16:06][INFO][BKUP] find obsolete backups: find /pg/backup/ -maxdepth 1 -type f -mmin +1200 -name 'backup*.lz4'[2025-07-13 06:16:06][WARN][BKUP] remove obsolete backups:
[2025-07-13 06:16:06][INFO][RMBK] remove old backup complete[2025-07-13 06:16:06][INFO][LOCK] release lock @ /tmp/backup.lock
[2025-07-13 06:16:06][INFO][DONE] backup procedure complete!
[2025-07-13 06:16:06][INFO]================================================================
Backup are compressed with lz4, You can unzip and extract the tarball with the following command:
mkdir -p /tmp/data # extract backup to this directorycat /pg/backup/backup_pg-meta_20250713.tar.lz4 | unlz4 -d -c | tar -xC /tmp/data
Logical Backup
You can also use the pg_dump command to perform a logical backup.
Logical backups cannot be used for PITR (Point In Time Recovery),
but they are useful for migrating data between different major versions, or implement flexible data export logic.
Bootstrap from Repo
Now let’s say you have an existing cluster pg-meta, and want to FORK it as pg-meta2:
You’ll need to create the new pg-meta2 cluster fork, then run pitr on it.
11.15.5 - Restore
Restore PostgreSQL from Backup
You can use the pre-configured pgbackrest to perform Point-in-Time Recovery (PITR) in Pigsty.
Manually: PITR with the pg-pitr hint script, do it manually, more flexible with more complexity.
Playbook: PITR with the pgsql-pitr.yml playbook, automatic, but less flexible and more error-prone.
If you are very convenient with your configuration, you can use the fully automatic playbook,
otherwise, consider do it step by step manually
Quick Start
If you want to roll back the pg-meta cluster to the previous timepoint, adding the pg_pitr:
pg-meta:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-meta2pg_pitr:{time:'2025-07-13 10:00:00+00'}# restore from the latest backup
Then run the pgsql-pitr.yml playbook, it will roll back the pg-meta cluster to the specified timepoint.
./pgsql-pitr.yml -l pg-meta
Restore PITR
The archive_mode will be disabled on recovered cluster to prevent unwanted WAL writes.
If the recovered database status is ok, you can enable the archive_mode and make a full backup.
postgres @ pg-meta $
psql -c 'ALTER SYSTEM RESET archive_mode; SELECT pg_reload_conf();'pg-backup full # take a new full backup
Recovery Target
You can specify different types of recovery targets in pg_pitr, but they are mutually exclusive:
name: restore to a named restore point (created by pg_create_restore_point)
xid: restore to a specific transaction ID (TXID/XID)
lsn: restore to a specific LSN (Log Sequence Number) point
The recovery type will be set accordingly if any of the above parameters is specified,
otherwise it will be set to latest (the end of the WAL archive stream).
The special immediate type can be used to instruct pgbackrest to minimize the recovery time by stop at the first consistent point.
Target Type
pg_pitr:{}# restore to the latest status (wal archive stream end)
pg_pitr:{time:"2025-07-13 10:00:00+00"}
pg_pitr:{lsn:"0/4001C80"}
pg_pitr:{xid:"250000"}
pg_pitr:{name:"some_restore_point"}
pg_pitr:{type:"immediate"}
By Time
The most frequently used target is the time point; you can specify the time point to restore to:
If you have a transaction that accidentally deleted some data, the best way to recover is to restore the database to the state before that transaction.
You can find the exact transaction id from monitoring dashboard, or find it from TXID from the CSVLOG.
Inclusive vs Exclusive
The target parameter is “inclusive” by default, which means the recovery will include the target point.
The exclusive flag will exclude that exact target, like the xid 24999 will be the last transaction being replayed
PostgreSQL uses the LSN (Log Sequence Number) to identify the position of a WAL record.
You can find it everywhere, like the PG LSN panel from Pigsty dashboards.
To restore to an exact point in the WAL stream, you may also specify the timeline parameter (default to latest)
Recovery Source
cluster: which cluster to restore? the current pg_cluster will be used by default, you can use any other cluster in the same pgbackrest repo
repo: overwrite the backup repo, use the same format in pgbackrest_repo
set: the latest backup set is used by default, but you can specify a specific pgbackrest backup by label
Pigsty will recover from the pgbackrest backup repository, if you are using a centralized backup repo (like MinIO/S3),
you can specify another “stanza” (another cluster’s backup directory) to restore from.
pg-meta2:hosts:{10.10.10.11:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-meta2pg_pitr:{cluster:pg-meta } # restore from the pg-meta cluster backup
The above configuration will mark the PITR procedure to use the pg-meta stanza.
You can also pass the pg_pitr parameter via CLI args:
This approach is semi-automatic, you will participate in the PITR process to make key decisions.
For example, this configuration will restore the pg-meta cluster itself to the specified timepoint
pg-meta:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-meta2pg_pitr:{time:'2025-07-13 10:00:00+00'}# restore from the latest backup
Let’s do this one step by step:
./pgsql-pitr.yml -l pg-meta -t down # pause patroni HA./pgsql-pitr.yml -l pg-meta -t pitr # run the pitr procedure./pgsql-pitr.yml -l pg-meta -t up # generate pgbackrest config and restore script
# down : # stop ha and shutdown patroni and postgres# - pause : # pause patroni auto failover# - stop : # stop patroni and postgres service# - stop_patroni : # stop patroni service# - stop_postgres : # stop postgres service# pitr : # perform the PITR procedure# - config : # generate pgbackrest config and restore script# - restore : # run the pgbackrest restore command# - recovery : # start postgres and complete recovery# - verify : # verify the recovered cluster control data# up: : # start postgres / patroni and resume ha# - etcd : # clean up etcd metadata before launching# - start : # start patroni and postgres service# - start_postgres : # start postgres service# - start_patroni : # start patroni service# - resume : # resume patroni auto failover
PITR Definition
There are more options available in the pg_pitr parameter:
pg_pitr:# define a PITR taskcluster:"some_pg_cls_name"# Source cluster nametype:latest # Recovery target type: time, xid, name, lsn, immediate, latesttime:"2025-01-01 10:00:00+00"# Recovery target: time, exclusive with xid, name, lsnname:"some_restore_point"# Recovery target: named restore point, exclusive with time, xid, lsnxid:"100000"# Recovery target: transaction ID, exclusive with time, name, lsnlsn:"0/3000000"# Recovery target: log sequence number, exclusive with time, name, xidtimeline:latest # Target timeline, can be an integer, latest by default,exclusive:false# Exclude the target point, default false?action:pause # Post-recovery action: pause, promote, shutdownarchive:false# Preserve archive settings? false by defaultdb_exclude:[template0, template1 ]db_include:[]link_map:pg_wal:'/data/wal'pg_xact:'/data/pg_xact'process:4# Parallel restore processesrepo:{}# Repository to restore fromdata:/pg/data # where to restore the dataport:5432# listen port of the recovered instance
11.15.6 - Example
Perform PITR manually in sandbox according to hint script
You can do PITR with the pgsql-pitr playbook, while in some case, you may want to perform PITR manually.
We’ll illustrate the procedure with the 4-node sandbox cluster with minio backup repo.
Init Sandbox
Prepare the 4-node sandbox environment with vagrant or terraform, then:
Now operate as the admin user (or dbsu) on the admin node to proceed.
Check Backup
To check the backup status, you’ll need to switch to the postgres user and use the pb command:
sudo su - postgres # switch to the dbsu: postgres userpb info # print pgbackrest backup info
The pb is the alias for pgbackrest, with auto scraped stanza name from pgbackrest config.
/etc/profile.d/pg-alias.sh
function pb(){localstanza=$(grep -o '\[[^][]*]' /etc/pgbackrest/pgbackrest.conf | head -n1 | sed 's/.*\[\([^]]*\)].*/\1/') pgbackrest --stanza=$stanza$@}
You can see the initial backup info, which is a full backup created at
The backup finish at 2025-07-13 02:27:33+00, this is the earliest time you can restore to.
Since wal archive is active, you can restore to any point in time after the backup, until the WAL end (now).
Generate Heartbeat
You can generate some heartbeat to simulate the workload. the /pg-bin/pg-heartbeat is for this purpose,
It will write a heartbeat timestamp to the monitor.heartbeat table every second.
make rh # run heartbeat: ssh 10.10.10.10 'sudo -iu postgres /pg/bin/pg-heartbeat'
while true;do pgbench -nv -P1 -c4 --rate=64 -T10 postgres://dbuser_meta:[email protected]:5433/meta;donepgbench (17.5 (Homebrew), server 17.4 (Ubuntu 17.4-1.pgdg24.04+2))progress: 1.0 s, 60.9 tps, lat 7.295 ms stddev 4.219, 0 failed, lag 1.818 ms
progress: 2.0 s, 69.1 tps, lat 6.296 ms stddev 1.983, 0 failed, lag 1.397 ms
...
PITR Manual
Now let’s choose a time point to recovery, let’s say 2025-07-13 03:03:03+00, which is a timepoint after the initial backup (and heartbeat).
To perform the manual PITR, use the pg-pitr util:
$ pg-pitr -t "2025-07-13 03:03:00+00"
It will generate the instructions for you to perform the recovery, it usually takes four steps:
Perform time PITR on pg-meta
[1. Stop PostgreSQL]=========================================== 1.1 Pause Patroni (if there are any replicas) $ pg pause <cls> # pause patroni auto failover 1.2 Shutdown Patroni
$ pt-stop # sudo systemctl stop patroni 1.3 Shutdown Postgres
$ pg-stop # pg_ctl -D /pg/data stop -m fast[2. Perform PITR]=========================================== 2.1 Restore Backup
$ pgbackrest --stanza=pg-meta --type=time --target='2025-07-13 03:03:00+00' restore
2.2 Start PG to Replay WAL
$ pg-start # pg_ctl -D /pg/data start 2.3 Validate and Promote
- If database content is ok, promote it to finish recovery, otherwise goto 2.1
$ pg-promote # pg_ctl -D /pg/data promote
[3. Restore Primary]=========================================== 3.1 Enable Archive Mode (Restart Required) $ psql -c 'ALTER SYSTEM SET archive_mode = on;' 3.1 Restart Postgres to Apply Changes
$ pg-restart # pg_ctl -D /pg/data restart 3.3 Restart Patroni
$ pt-restart # sudo systemctl restart patroni[4. Restore Cluster]=========================================== 4.1 Re-Init All [**REPLICAS**](if any) - 4.1.1 option 1: restore replicas with same pgbackrest cmd (require central backup repo) $ pgbackrest --stanza=pg-meta --type=time --target='2025-07-13 03:03:00+00' restore
- 4.1.2 option 2: nuke the replica data dir and restart patroni (may take long time to restore) $ rm -rf /pg/data/*; pt-restart
- 4.1.3 option 3: reinit with patroni, which may fail if primary lsn < replica lsn
$ pg reinit pg-meta
4.2 Resume Patroni
$ pg resume pg-meta
4.3 Full Backup (optional) $ pg-backup full # IT's recommend to make a new full backup after PITR
Single-Node Example
Let’s start with the simple 1-node pg-meta cluster as an example, which is simpler.
We don’t want patroni HA to take over until we are sure the data is correct, so we start postgres manually:
pg-start
waiting for server to start....2025-07-13 03:19:33.133 UTC [39294] LOG: redirecting log output to logging collector process
2025-07-13 03:19:33.133 UTC [39294] HINT: Future log output will appear in directory "/pg/log/postgres".
doneserver started
Now you can check the data to see if the it is at the timepoint you want.
You can validate it by checking some latest timestamp from business tables, or in this case, check via the heartbeat table.
The timestamp is right before the timepoint we specified! (2025-07-13 03:03:00+00).
If this is not the timepoint you want, you can repeat the restore with a different timepoint.
It’s rapid since recovery is performed in an incremental and parallel way.
It’s ok to retry until you get the right point.
Promote Leader
The recovered postgres cluster is in recovery mode, so it will reject any write operations until you promote it to primary.
These recovery params are generated by pgBackRest in the config file.
/pg/data/postgresql.auto.conf
postgres@pg-meta-1:~$ cat /pg/data/postgresql.auto.conf# Do not edit this file or use ALTER SYSTEM manually!# It is managed by Pigsty & Ansible automatically!# Recovery settings generated by pgBackRest restore on 2025-07-13 03:17:08archive_mode='off'restore_command='pgbackrest --stanza=pg-meta archive-get %f "%p"'recovery_target_time='2025-07-13 03:03:00+00'
If data is correct, you can promote it to primary, mark it as the new leader and ready to accept writes.
pg-promote
waiting for server to promote.... doneserver promoted
psql -c 'SELECT pg_is_in_recovery()'# the 'f' means it is promoted to primary pg_is_in_recovery
-------------------
f
(1 row)
New Timeline and Split Brain
Once promoted, the database cluster will enter a new timeline (the leader epoch).
If there’s any write traffic, it will be written to the new timeline.
Restore Cluster
Finally, it’s not only the data that need recovery, but also the cluster state, such as:
patroni takeover
archive mode
backup set
replicas
Patroni Takeover
You postgres is start directly, to restore HA takeover; you’ll have to start the patroni service:
pt-start # sudo systemctl start patroni
pg resume pg-meta # resume patroni auto failover (if you have paused it before)
Archive Mode
The archive_mode is disabled by pgbackrest during recovery。
If you want the new leader’s writes to be archived in the backup repo, you also need to enable the archive_mode config.
psql -c 'show archive_mode' archive_mode
--------------
off
# you can also edit the postgresql.auto.conf directly and reload with pg_ctlsed -i '/archive_mode/d' /pg/data/postgresql.auto.conf
pg_ctl -D /pg/data reload
Backup Set
It’s usually a good idea to take a new full backup after PITR, but it’s optional.
Replicas
If your postgres cluster has replicas, you’ll need to perform the PITR on each replica as well.
Or, the simple way is to nuke the replica data directory and restart patroni, which will re-initialize the replica from the primary.
We will cover this case in the next multi-node cluster example.
Multi-Node Example
Now let’s play with the 3-node pg-test cluster as an PITR example.
11.16 - Kernel
Replace vanilla PostgreSQL with exotic kernel forks
Pigsty supports various PostgreSQL kernels and compatible forks,
enabling you to simulate different database systems while leveraging PostgreSQL’s ecosystem.
Each kernel provides unique capabilities and compatibility layers.
Database Kernels
PostgreSQL
Vanilla Postgres with 437 Extensions
Citus
Native Distributive Extension
Babelfish
SQL Server wire-compatible
IvorySQL
Oracle grammar & PL/SQL compatible
OpenHalo
MySQL wire-compatibility
Percona
Transparent Data Encryption
OrioleDB
OLTP-optimized cloud-native storage engine
PolarDB PG
Aurora-like RAC with china domestic compliance
Supabase
Backend as a Service, self-hosting Firebase
FerretDB
Mongo Wire-Compatibility over PostgreSQL
Choose the Right Kernel
Note
Flexible Kernel: Choose the right kernel for your specific use case - whether you need MSSQL compatibility, Oracle features, or horizontal scaling capabilities.
Citus transforms PostgreSQL into a distributed database system, enabling horizontal scaling across multiple nodes.
Deploy native HA Citus clusters with Pigsty for better throughput and performance.
Key Features
Distributed Tables: Automatically shard tables across worker nodes
Distributed Queries: Execute queries across the entire cluster
High Availability: Built-in replication and failover capabilities
Real-time Analytics: Handle both transactional and analytical workloads
Postgres Compatibility: Maintain full PostgreSQL feature compatibility
Self-host Supabase with existing managed HA PostgreSQL clusters, launching the stateless components with docker-compose for a complete Firebase alternative.
Key Features
Real-time APIs: Auto-generated REST and GraphQL APIs
Real-time Subscriptions: WebSocket-based real-time data sync
Authentication: Built-in user authentication and authorization
Storage: File storage with CDN capabilities
Edge Functions: Serverless functions for custom logic
Use Cases
Rapid application development with backend-as-a-service
Real-time applications requiring instant data sync
JAMstack applications needing serverless backend
Mobile and web apps require authentication and storage
Note
Full Stack: Provides a complete backend solution with PostgreSQL as the foundation.
pgsql : the minimal postgres kernel config example (THIS CONFIG)
Configure
Nothing special needs to be tuned for vanilla PostgreSQL kernel:
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_users:- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin ] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer }pg_databases:- {name:meta, baseline:cmdb.sql ,comment:pigsty meta database ,schemas:[pigsty] ,extensions:[vector ]}pg_hba_rules:- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1ampg_packages:[pgsql-main, pgsql-common ] # pg kernel and common utils#pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
To use a different PostgreSQL major version, you can configure with -v parameter:
./configure -c pgsql # the default is postgresql 18./configure -c pgsql -v 17# use postgresql 17./configure -c pgsql -v 16# use postgresql 16./configure -c pgsql -v 15# use postgresql 15./configure -c pgsql -v 14# use postgresql 14./configure -c pgsql -v 13# use postgresql 13
If PostgreSQL cluster is already installed, you’ll need to uninstall it before installing the new version
./pgsql-rm.yml # -l pg-meta
PostgreSQL beta
To use PostgreSQL beta version (19 not released), you’ll also need to add the beta repo to your node_repo_modules
(or repo_modules if you are building a local repo)
The configure will do that for you if you are use the -v 19 argument:
./configure -c pgsql -v 19# use the postgresql 19 kernel (beta not released yet)./install.yml # setup everything with pigsty
Beware Pigsty is not build extensions for PostgreSQL 19 yet,
so only those existing extensions in the PGDG repo are currently available.
We will start building them after the PostgreSQL 19 is released.
Multi-Node
To setup a multi-node PostgreSQL cluster, you can check the PGSQL: Configure for details:
This is the example 3-node pg-test cluster in the full config template:
11.16.2 - Citus
Native Distributive Extension for PostgreSQL Sharding
Citus is a PostgreSQL extension that transforms PostgreSQL into a distributed database, enabling horizontal scaling across multiple nodes to handle large amounts of data and queries.
Since Patroni v3.0, native support for Citus high availability has been provided, simplifying the setup of Citus clusters. Pigsty also offers native support for this.
Pigsty v3.7.0 pins the Citus template to PostgreSQL 17; Citus packages are not available for PostgreSQL 18 in this release.
Citus Cluster
Pigsty natively supports Citus. Refer to conf/citus.yml.
This example uses a four-node sandbox with a Citus cluster named pg-citus, consisting of a two-node coordinator cluster pg-citus0 and two worker clusters pg-citus1 and pg-citus2.
pg-citus:hosts:10.10.10.10:{pg_group:0, pg_cluster:pg-citus0 ,pg_vip_address:10.10.10.2/24 ,pg_seq:1, pg_role:primary }10.10.10.11:{pg_group:0, pg_cluster:pg-citus0 ,pg_vip_address:10.10.10.2/24 ,pg_seq:2, pg_role:replica }10.10.10.12:{pg_group:1, pg_cluster:pg-citus1 ,pg_vip_address:10.10.10.3/24 ,pg_seq:1, pg_role:primary }10.10.10.13:{pg_group:2, pg_cluster:pg-citus2 ,pg_vip_address:10.10.10.4/24 ,pg_seq:1, pg_role:primary }vars:pg_mode:citus # pgsql cluster mode: cituspg_version:17# Citus is not available for PG18 in v3.7.0pg_shard:pg-citus # Citus shard name: pg-cituspg_primary_db:citus # primary database used by Cituspg_vip_enabled:true# enable VIP for Citus clusterpg_vip_interface:eth1 # VIP interface for all memberspg_dbsu_password:DBUser.Postgres # all DBSU passwords for Citus clusterpg_extensions:[citus, postgis, pgvector, topn, pg_cron, hll ] # install these extensionspg_libs:'citus, pg_cron, pg_stat_statements'# Citus will be added automatically by Patronipg_users:[{name:dbuser_citus ,password:DBUser.Citus ,pgbouncer:true ,roles:[dbrole_admin ] }]pg_databases:[{name:citus ,owner:dbuser_citus ,extensions:[citus, vector, topn, pg_cron, hll ] }]pg_parameters:cron.database_name:cituscitus.node_conninfo:'sslmode=require sslrootcert=/pg/cert/ca.crt sslmode=verify-full'pg_hba_rules:- {user:'all' ,db:all ,addr:127.0.0.1/32 ,auth:ssl ,title:'all user ssl access from localhost'}- {user:'all' ,db:all ,addr:intra ,auth:ssl ,title:'all user ssl access from intranet'}
Compared to a standard PostgreSQL cluster, Citus cluster configuration has some specific requirements. First, ensure that the Citus extension is downloaded, installed, loaded, and enabled. This involves the following four parameters:
repo_packages: Must include the citus extension, or you need to use a PostgreSQL offline package with the Citus extension.
pg_extensions: Must include the citus extension, meaning you need to install the citus extension on each node.
pg_libs: Must include the citus extension, and it must be first in the list, but now Patroni will automatically handle this.
pg_databases: Define a primary database with the citus extension installed.
Additionally, ensure the configuration for the Citus cluster is correct:
pg_mode: Must be set to citus to inform Patroni to use the Citus mode.
pg_primary_db: Specify the primary database name, which must have the citus extension (named citus here).
pg_shard: Specify a unified name as a prefix for all horizontal shard PG clusters (e.g., pg-citus).
pg_group: Specify a shard number, starting from zero for the coordinator cluster and incrementing for worker clusters.
pg_cluster: Must match the combination of [pg_shard] and [pg_group].
pg_dbsu_password: Set a non-empty plain-text password for proper Citus functionality.
pg_parameters: It is recommended to set the citus.node_conninfo parameter, which enforces SSL access and requires node-to-node client certificate verification.
Once configured, deploy the Citus cluster just like a regular PostgreSQL cluster using pgsql.yml.
Managing Citus Clusters
After defining the Citus cluster, use the same playbook pgsql.yml to deploy the Citus cluster:
./pgsql.yml -l pg-citus # Deploy Citus cluster pg-citus
Any DBSU user (postgres) can use patronictl (alias: pg) to list the status of the Citus cluster:
Each horizontal shard cluster can be treated as a separate PGSQL cluster, managed with the pg (patronictl) command. Note that when using pg to manage the Citus cluster, the --group parameter must be used to specify the cluster shard number:
pg list pg-citus --group 0# Use --group 0 to specify the shard number
Citus has a system table called pg_dist_node to record node information, which Patroni automatically maintains.
PGURL=postgres://postgres:[email protected]/citus
psql $PGURL -c 'SELECT * FROM pg_dist_node;'# View node information
Additionally, you can view user authentication information (restricted to superusers):
$ psql $PGURL -c 'SELECT * FROM pg_dist_authinfo;'# View node authentication info (superuser only)
You can then access the Citus cluster with regular business users (e.g., dbuser_citus with DDL permissions):
psql postgres://dbuser_citus:[email protected]/citus -c 'SELECT * FROM pg_dist_node;'
Using the Citus Cluster
When using a Citus cluster, we highly recommend reading the Citus Official Documentation to understand its architecture and core concepts.
Key to this is understanding the five types of tables in Citus, their characteristics, and use cases:
Distributed Table
Reference Table
Local Table
Local Management Table
Schema Table
On the coordinator node, you can create distributed and reference tables and query them from any data node. Since version 11.2, any Citus database node can act as a coordinator.
We can use pgbench to create some tables, distributing the main table (pgbench_accounts) across the nodes, and using other smaller tables as reference tables:
Production citus deployment usually requires physical replication for both coordinator and each worker cluster.
For example, in simu.yml there’s a 10-node cluster cluster:
pg-citus:# citus grouphosts:10.10.10.50:{pg_group:0, pg_cluster:pg-citus0 ,pg_vip_address:10.10.10.60/24 ,pg_seq:0, pg_role:primary }10.10.10.51:{pg_group:0, pg_cluster:pg-citus0 ,pg_vip_address:10.10.10.60/24 ,pg_seq:1, pg_role:replica }10.10.10.52:{pg_group:1, pg_cluster:pg-citus1 ,pg_vip_address:10.10.10.61/24 ,pg_seq:0, pg_role:primary }10.10.10.53:{pg_group:1, pg_cluster:pg-citus1 ,pg_vip_address:10.10.10.61/24 ,pg_seq:1, pg_role:replica }10.10.10.54:{pg_group:2, pg_cluster:pg-citus2 ,pg_vip_address:10.10.10.62/24 ,pg_seq:0, pg_role:primary }10.10.10.55:{pg_group:2, pg_cluster:pg-citus2 ,pg_vip_address:10.10.10.62/24 ,pg_seq:1, pg_role:replica }10.10.10.56:{pg_group:3, pg_cluster:pg-citus3 ,pg_vip_address:10.10.10.63/24 ,pg_seq:0, pg_role:primary }10.10.10.57:{pg_group:3, pg_cluster:pg-citus3 ,pg_vip_address:10.10.10.63/24 ,pg_seq:1, pg_role:replica }10.10.10.58:{pg_group:4, pg_cluster:pg-citus4 ,pg_vip_address:10.10.10.64/24 ,pg_seq:0, pg_role:primary }10.10.10.59:{pg_group:4, pg_cluster:pg-citus4 ,pg_vip_address:10.10.10.64/24 ,pg_seq:1, pg_role:replica }vars:pg_mode:citus # pgsql cluster mode: cituspg_version:17# Citus is not available for PG18 in v3.7.0pg_shard:pg-citus # citus shard name: pg-cituspg_primary_db:citus # primary database used by cituspg_vip_enabled:true# enable vip for citus clusterpg_vip_interface:eth1 # vip interface for all memberspg_dbsu_password:DBUser.Postgres # enable dbsu password access for cituspg_extensions:[citus, postgis, pgvector, topn, pg_cron, hll ] # install these extensionspg_libs:'citus, pg_cron, pg_stat_statements'# citus will be added by patroni automaticallypg_users:[{name:dbuser_citus ,password:DBUser.Citus ,pgbouncer:true ,roles:[dbrole_admin ] }]pg_databases:[{name:citus ,owner:dbuser_citus ,extensions:[citus, vector, topn, pg_cron, hll ] }]pg_parameters:cron.database_name:cituscitus.node_conninfo:'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'pg_hba_rules:- {user:'all' ,db:all ,addr:127.0.0.1/32 ,auth:ssl ,title:'all user ssl access from localhost'}- {user:'all' ,db:all ,addr:intra ,auth:ssl ,title:'all user ssl access from intranet'}
We’ll cover a range of advanced topics in subsequent tutorials:
Read-write separation
Failover handling
Consistent backup and restore
Advanced monitoring and troubleshooting
Connection pool
11.16.3 - Babelfish
MS SQL Server Wire compatibility on PostgreSQL
Pigsty allows users to create a Microsoft SQL Server compatible PostgreSQL cluster using Babelfish and WiltonDB!
Babelfish: An open-source MSSQL (Microsoft SQL Server) compatibility extension Open Sourced by AWS
WiltonDB: A PostgreSQL kernel distribution focusing on integrating Babelfish
Babelfish is a PostgreSQL extension, but it works on a slightly modified PostgreSQL kernel Fork, WiltonDB provides compiled kernel binaries and extension binary packages on EL/Ubuntu systems.
Pigsty can replace the native PostgreSQL kernel with WiltonDB, providing an out-of-the-box MSSQL compatible cluster along with all the supported by common PostgreSQL clusters, such as HA, PITR, IaC, monitoring, etc.
WiltonDB is very similar to PostgreSQL 15, but it can not use vanilla PostgreSQL extensions directly. WiltonDB has several re-compiled extensions such as system_stats, pg_hint_plan and tds_fdw.
The cluster will listen on the default PostgreSQL port and the default MSSQL 1433 port, providing MSSQL services via the TDS WireProtocol on this port. You can connect to the MSSQL service provided by Pigsty using any MSSQL client, such as SQL Server Management Studio, or using the sqlcmd command-line tool.
curl -fsSL https://repo.pigsty.io/get | bash -s v3.7.0;cd ~/pigsty;./configure -c mssql # Use mssql (babelfish) template./install.yml # install everything with pigsty
For production deployments, make sure to modify the password parameters in the pigsty.yml config before running the install playbook.
Notes
When installing and deploying the MSSQL module, please pay special attention to the following points:
WiltonDB is available on EL (7/8/9) and Ubuntu (20.04/22.04) but not available on Debian systems.
WiltonDB is currently compiled based on PostgreSQL 15, so you need to specify pg_version: 15.
On EL systems, the wiltondb binary is installed by default in the /usr/bin/ directory, while on Ubuntu systems, it is installed in the /usr/lib/postgresql/15/bin/ directory, which is different from the official PostgreSQL binary location.
In WiltonDB compatibility mode, the HBA password authentication rule needs to use md5 instead of scram-sha-256. Therefore, you need to override Pigsty’s default HBA rule set and insert the md5 authentication rule required by SQL Server before the dbrole_readonly wildcard authentication rule.
WiltonDB can only be enabled for a primary database, and you should designate a user as the Babelfish superuser, allowing Babelfish to create databases and users. The default is mssql and dbuser_myssql. If you change this, you should also modify the user in files/mssql.sql.
The WiltonDB TDS cable protocol compatibility plugin babelfishpg_tds needs to be enabled in shared_preload_libraries.
After enabling the WiltonDB extension, it listens on the default MSSQL port 1433. You can override Pigsty’s default service definitions to redirect the primary and replica services to port 1433 instead of the 5432 / 6432ports.
The following parameters need to be configured for the MSSQL database cluster:
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_users:- {name:dbuser_mssql ,password:DBUser.MSSQL ,superuser:true, pgbouncer:true ,roles:[dbrole_admin], comment:superuser & owner for babelfish }pg_databases:- name:mssqlbaseline:mssql.sqlextensions:[uuid-ossp, babelfishpg_common, babelfishpg_tsql, babelfishpg_tds, babelfishpg_money, pg_hint_plan, system_stats, tds_fdw]owner:dbuser_mssqlparameters:{'babelfishpg_tsql.migration_mode' :'multi-db'}comment:babelfish cluster, a MSSQL compatible pg clusternode_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1am# Babelfish / WiltonDB Ad Hoc Settingspg_mode:mssql # Microsoft SQL Server Compatible Modepg_version:15pg_packages:[wiltondb, pgsql-common, sqlcmd ]pg_libs:'babelfishpg_tds, pg_stat_statements, auto_explain'# add timescaledb to shared_preload_librariespg_default_hba_rules:# overwrite default HBA rules for babelfish cluster- {user:'${dbsu}',db:all ,addr:local ,auth:ident ,title:'dbsu access via local os user ident'}- {user:'${dbsu}',db:replication ,addr:local ,auth:ident ,title:'dbsu replication from local os ident'}- {user:'${repl}',db:replication ,addr:localhost ,auth:pwd ,title:'replicator replication from localhost'}- {user:'${repl}',db:replication ,addr:intra ,auth:pwd ,title:'replicator replication from intranet'}- {user:'${repl}',db:postgres ,addr:intra ,auth:pwd ,title:'replicator postgres db from intranet'}- {user:'${monitor}',db:all ,addr:localhost ,auth:pwd ,title:'monitor from localhost with password'}- {user:'${monitor}',db:all ,addr:infra ,auth:pwd ,title:'monitor from infra host with password'}- {user:'${admin}',db:all ,addr:infra ,auth:ssl ,title:'admin @ infra nodes with pwd & ssl'}- {user:'${admin}',db:all ,addr:world ,auth:ssl ,title:'admin @ everywhere with ssl & pwd'}- {user:dbuser_mssql ,db:mssql ,addr:intra ,auth:md5 ,title:'allow mssql dbsu intranet access'}# <--- use md5 auth method for mssql user- {user:'+dbrole_readonly',db:all ,addr:localhost ,auth:pwd ,title:'pgbouncer read/write via local socket'}- {user:'+dbrole_readonly',db:all ,addr:intra ,auth:pwd ,title:'read/write biz user via password'}- {user:'+dbrole_offline' ,db:all ,addr:intra ,auth:pwd ,title:'allow etl offline tasks from intranet'}pg_default_services:# route primary & replica service to mssql port 1433- {name:primary ,port:5433 ,dest:1433 ,check:/primary ,selector:"[]"}- {name:replica ,port:5434 ,dest:1433 ,check:/read-only ,selector:"[]", backup:"[? pg_role == `primary` || pg_role == `offline` ]"}- {name:default ,port:5436 ,dest:postgres ,check:/primary ,selector:"[]"}- {name:offline ,port:5438 ,dest:postgres ,check:/replica ,selector:"[? pg_role == `offline` || pg_offline_query ]", backup:"[? pg_role == `replica` && !pg_offline_query]"}
You can define business databases & users in the pg_databases and pg_users section:
#----------------------------------## pgsql (singleton on current node)#----------------------------------## this is an example single-node postgres cluster with postgis & timescaledb installed, with one biz database & two biz userspg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }# <---- primary instance with read-write capabilityvars:pg_cluster:pg-testpg_users:# create MSSQL superuser- {name:dbuser_mssql ,password:DBUser.MSSQL ,superuser:true, pgbouncer:true ,roles:[dbrole_admin], comment:superuser & owner for babelfish }pg_primary_db:mssql # use `mssql` as the primary sql server databasepg_databases:- name:mssqlbaseline:mssql.sql # init babelfish database & userextensions:- {name:uuid-ossp }- {name:babelfishpg_common }- {name:babelfishpg_tsql }- {name:babelfishpg_tds }- {name:babelfishpg_money }- {name:pg_hint_plan }- {name:system_stats }- {name:tds_fdw }owner:dbuser_mssqlparameters:{'babelfishpg_tsql.migration_mode' :'multi-db'}comment:babelfish cluster, a MSSQL compatible pg cluster
Client Access
You can use any SQL Server compatible client tool to access this database cluster.
Microsoft provides sqlcmd as the official command-line tool.
Besides, they have a go version cli tool: go-sqlcmd
Install go-sqlcmd:
curl -LO https://github.com/microsoft/go-sqlcmd/releases/download/v1.4.0/sqlcmd-v1.4.0-linux-amd64.tar.bz2
tar xjvf sqlcmd-v1.4.0-linux-amd64.tar.bz2
sudo mv sqlcmd* /usr/bin/
Get started with go-sqlcmd
$ sqlcmd -S 10.10.10.10,1433 -U dbuser_mssql -P DBUser.MSSQL
1> select @@version
2> go
version
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Babelfish for PostgreSQL with SQL Server Compatibility - 12.0.2000.8
Oct 222023 17:48:32
Copyright (c) Amazon Web Services
PostgreSQL 15.4 (EL 1:15.4.wiltondb3.3_2-2.el8) on x86_64-redhat-linux-gnu (Babelfish 3.3.0)(1 row affected)
You can route service traffic to MSSQL 1433 port instead of 5433/5434:
# route 5433 on all members to 1433 on primarysqlcmd -S 10.10.10.11,5433 -U dbuser_mssql -P DBUser.MSSQL
# route 5434 on all members to 1433 on replicassqlcmd -S 10.10.10.11,5434 -U dbuser_mssql -P DBUser.MSSQL
Install
If you have the Internet access, you can add the WiltonDB repository to the node and install it as a node package directly:
It’s OK to install vanilla PostgreSQL and WiltonDB on the same node, but you can only run one of them at a time, and this is not recommended for production environments.
Extensions
Most of the PGSQL module’s extensions (non-SQL class) cannot be used directly on the WiltonDB core of the MSSQL module and need to be recompiled.
WiltonDB currently comes with the following extension plugins:
Name
Version
Comment
dblink
1.2
connect to other PostgreSQL databases from within a database
adminpack
2.1
administrative functions for PostgreSQL
dict_int
1.0
text search dictionary template for integers
intagg
1.1
integer aggregator and enumerator (obsolete)
dict_xsyn
1.0
text search dictionary template for extended synonym processing
amcheck
1.3
functions for verifying relation integrity
autoinc
1.0
functions for autoincrementing fields
bloom
1.0
bloom access method - signature file based index
fuzzystrmatch
1.1
determine similarities and distance between strings
intarray
1.5
functions, operators, and index support for 1-D arrays of integers
btree_gin
1.3
support for indexing common datatypes in GIN
btree_gist
1.7
support for indexing common datatypes in GiST
hstore
1.8
data type for storing sets of (key, value) pairs
hstore_plperl
1.0
transform between hstore and plperl
isn
1.2
data types for international product numbering standards
hstore_plperlu
1.0
transform between hstore and plperlu
jsonb_plperl
1.0
transform between jsonb and plperl
citext
1.6
data type for case-insensitive character strings
jsonb_plperlu
1.0
transform between jsonb and plperlu
jsonb_plpython3u
1.0
transform between jsonb and plpython3u
cube
1.5
data type for multidimensional cubes
hstore_plpython3u
1.0
transform between hstore and plpython3u
earthdistance
1.1
calculate great-circle distances on the surface of the Earth
lo
1.1
Large Object maintenance
file_fdw
1.0
foreign-data wrapper for flat file access
insert_username
1.0
functions for tracking who changed a table
ltree
1.2
data type for hierarchical tree-like structures
ltree_plpython3u
1.0
transform between ltree and plpython3u
pg_walinspect
1.0
functions to inspect contents of PostgreSQL Write-Ahead Log
moddatetime
1.0
functions for tracking last modification time
old_snapshot
1.0
utilities in support of old_snapshot_threshold
pgcrypto
1.3
cryptographic functions
pgrowlocks
1.2
show row-level locking information
pageinspect
1.11
inspect the contents of database pages at a low level
pg_surgery
1.0
extension to perform surgery on a damaged relation
seg
1.4
data type for representing line segments or floating-point intervals
pgstattuple
1.5
show tuple-level statistics
pg_buffercache
1.3
examine the shared buffer cache
pg_freespacemap
1.2
examine the free space map (FSM)
postgres_fdw
1.1
foreign-data wrapper for remote PostgreSQL servers
pg_prewarm
1.2
prewarm relation data
tcn
1.0
Triggered change notifications
pg_trgm
1.6
text similarity measurement and index searching based on trigrams
xml2
1.1
XPath querying and XSLT
refint
1.0
functions for implementing referential integrity (obsolete)
pg_visibility
1.2
examine the visibility map (VM) and page-level visibility info
pg_stat_statements
1.10
track planning and execution statistics of all SQL statements executed
sslinfo
1.2
information about SSL certificates
tablefunc
1.0
functions that manipulate whole tables, including crosstab
tsm_system_rows
1.0
TABLESAMPLE method which accepts number of rows as a limit
tsm_system_time
1.0
TABLESAMPLE method which accepts time in milliseconds as a limit
unaccent
1.1
text search dictionary that removes accents
uuid-ossp
1.1
generate universally unique identifiers (UUIDs)
plpgsql
1.0
PL/pgSQL procedural language
babelfishpg_money
1.1.0
babelfishpg_money
system_stats
2.0
EnterpriseDB system statistics for PostgreSQL
tds_fdw
2.0.3
Foreign data wrapper for querying a TDS database (Sybase or Microsoft SQL Server)
babelfishpg_common
3.3.3
Transact SQL Datatype Support
babelfishpg_tds
1.0.0
TDS protocol extension
pg_hint_plan
1.5.1
babelfishpg_tsql
3.3.1
Transact SQL compatibility
11.16.4 - IvorySQL
PostgreSQL fork with oracle (grammar) compatibility
IvorySQL is an open-source “Oracle-compatible” PostgreSQL kernel, developed by HighGo, licensed under Apache 2.0.
The Oracle compatibility here refers to compatibility at the PL/SQL, syntax, built-in functions, data types, system views, MERGE, and GUC parameter levels.
It’s not a wire protocol compatibility like Babelfish, openHalo, or FerretDB that allows using the original client drivers.
Users still need to use PostgreSQL client tools to access IvorySQL, but can use Oracle-compatible syntax.
Currently, IvorySQL’s latest version 5.0 maintains compatibility with PostgreSQL’s latest minor version 18.0,
and provides binary RPM/DEB packages for mainstream Linux distributions. Pigsty offers the option to replace the native PostgreSQL with the IvorySQL kernel in PG RDS.
Quick Start
Use the standard procedure to install Pigsty with the ivory configuration template:
curl -fsSL https://repo.pigsty.io/get | bash -s v3.7.0;cd ~/pigsty;./configure -c ivory # Use IvorySQL configuration template./install.yml # Run installation playbook
For production deployments, you should edit the auto-generated pigsty.yml configuration file to modify parameters like passwords before executing ./install.yml for deployment.
The latest IvorySQL 5.0 is equivalent to PostgreSQL 18.0 Any client tool compatible with PostgreSQL’s wire protocol can access IvorySQL clusters.
By default, you can use a PostgreSQL client to access through the alternative 1521 port, which enables Oracle compatibility mode by default.
Configuration Instructions
To use the IvorySQL kernel in Pigsty, modify the following four configuration parameters:
It’s that simple — just add these four lines to the global variables in the configuration file, and Pigsty will replace the native PostgreSQL kernel with IvorySQL:
Most of the PGSQL modules’ extension (non-SQL classes) cannot be used directly on the IvorySQL kernel.
If you need to use them, you need to recompile and install from source code for the new kernel.
Caveats
The IvorySQL software package is located in the pigsty-infra repository, not in pigsty-pgsql or pigsty-ivory repositories.
Pigsty does not assume any warranty for using the IvorySQL kernel, and any issues or requests should be addressed to the manufacturer.
11.16.5 - Percona
Percona Postgres Distribution with TDE support
Percona Postgres is a patched Postgres kernel with pg_tde (Transparent Data Encryption) extension.
It is compatible with PostgreSQL 18.1, and available on all supported platforms in Pigsty.
curl -fsSL https://repo.pigsty.io/get | bash -s v3.7.0;cd ~/pigsty;./configure -c pgtde # use percona postgres kernel./install.yml # setup everything with pigsty
Configure
The following parameters need to be tuned to deploy a percona cluster:
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_users:- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin ] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer }pg_databases:- name:metabaseline:cmdb.sqlcomment:pigsty tde databaseschemas:[pigsty]extensions:[vector, postgis, pg_tde ,pgaudit, { name: pg_stat_monitor, schema: monitor } ]pg_hba_rules:- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1am# Percona PostgreSQL TDE Ad Hoc Settingspg_packages:[percona-main, pgsql-common ] # install percona postgres packagespg_libs:'pg_tde, pgaudit, pg_stat_statements, pg_stat_monitor, auto_explain'
Extensions
Percona Postgres has 80 available extensions, including pg_tde, pgvector, postgis, pgaudit, set_user, pg_stat_monitor,….
name
version
comment
hstore_plperlu
1.0
transform between hstore and plperlu
jsonb_plperl
1.0
transform between jsonb and plperl
intagg
1.1
integer aggregator and enumerator (obsolete)
pltcl
1.0
PL/Tcl procedural language
isn
1.3
data types for international product numbering standards
pgstattuple
1.5
show tuple-level statistics
postgis_topology-3
3.5.4
PostGIS topology spatial types and functions
postgis_raster
3.5.4
PostGIS raster types and functions
tsm_system_rows
1.0
TABLESAMPLE method which accepts number of rows as a limit
lo
1.2
Large Object maintenance
hstore_plperl
1.0
transform between hstore and plperl
ltree
1.3
data type for hierarchical tree-like structures
postgis_raster-3
3.5.4
PostGIS raster types and functions
postgis_topology
3.5.4
PostGIS topology spatial types and functions
pgrowlocks
1.2
show row-level locking information
address_standardizer_data_us-3
3.5.4
Address Standardizer US dataset example
uuid-ossp
1.1
generate universally unique identifiers (UUIDs)
postgis-3
3.5.4
PostGIS geometry and geography spatial types and functions
hstore_plpython3u
1.0
transform between hstore and plpython3u
postgis
3.5.4
PostGIS geometry and geography spatial types and functions
set_user
4.2.0
similar to SET ROLE but with added logging
postgis_tiger_geocoder-3
3.5.4
PostGIS tiger geocoder and reverse geocoder
jsonb_plperlu
1.0
transform between jsonb and plperlu
pg_surgery
1.0
extension to perform surgery on a damaged relation
xml2
1.2
XPath querying and XSLT
pg_stat_monitor
2.3
The pg_stat_monitor is a PostgreSQL Query Performance Monitoring tool, based on PostgreSQL contrib module pg_stat_statements. pg_stat_monitor provides aggregated statistics, client information, plan details including plan, and histogram information.
pg_tde
2.1
pg_tde access method
plpgsql
1.0
PL/pgSQL procedural language
address_standardizer-3
3.5.4
Used to parse an address into constituent elements. Generally used to support geocoding address normalization step.
tablefunc
1.0
functions that manipulate whole tables, including crosstab
hstore
1.8
data type for storing sets of (key, value) pairs
vector
0.8.1
vector data type and ivfflat and hnsw access methods
postgis_tiger_geocoder
3.5.4
PostGIS tiger geocoder and reverse geocoder
dblink
1.2
connect to other PostgreSQL databases from within a database
pltclu
1.0
PL/TclU untrusted procedural language
pg_trgm
1.6
text similarity measurement and index searching based on trigrams
sslinfo
1.2
information about SSL certificates
pg_stat_statements
1.12
track planning and execution statistics of all SQL statements executed
bool_plperlu
1.0
transform between bool and plperlu
cube
1.5
data type for multidimensional cubes
ltree_plpython3u
1.0
transform between ltree and plpython3u
amcheck
1.5
functions for verifying relation integrity
postgis_sfcgal
3.5.4
PostGIS SFCGAL functions
plpython3u
1.0
PL/Python3U untrusted procedural language
tsm_system_time
1.0
TABLESAMPLE method which accepts time in milliseconds as a limit
intarray
1.5
functions, operators, and index support for 1-D arrays of integers
btree_gist
1.8
support for indexing common datatypes in GiST
plperlu
1.0
PL/PerlU untrusted procedural language
fuzzystrmatch
1.2
determine similarities and distance between strings
bool_plperl
1.0
transform between bool and plperl
btree_gin
1.3
support for indexing common datatypes in GIN
pg_prewarm
1.2
prewarm relation data
pg_repack
1.5.3
Reorganize tables in PostgreSQL databases with minimal locks
citext
1.8
data type for case-insensitive character strings
pgcrypto
1.4
cryptographic functions
moddatetime
1.0
functions for tracking last modification time
plperl
1.0
PL/Perl procedural language
seg
1.4
data type for representing line segments or floating-point intervals
earthdistance
1.2
calculate great-circle distances on the surface of the Earth
unaccent
1.1
text search dictionary that removes accents
postgres_fdw
1.2
foreign-data wrapper for remote PostgreSQL servers
pg_logicalinspect
1.0
functions to inspect logical decoding components
tcn
1.0
Triggered change notifications
bloom
1.0
bloom access method - signature file based index
dict_int
1.0
text search dictionary template for integers
autoinc
1.0
functions for autoincrementing fields
address_standardizer_data_us
3.5.4
Address Standardizer US dataset example
postgis_sfcgal-3
3.5.4
PostGIS SFCGAL functions
jsonb_plpython3u
1.0
transform between jsonb and plpython3u
file_fdw
1.0
foreign-data wrapper for flat file access
pgaudit
18.0
provides auditing functionality
dict_xsyn
1.0
text search dictionary template for extended synonym processing
pg_walinspect
1.1
functions to inspect contents of PostgreSQL Write-Ahead Log
pg_buffercache
1.6
examine the shared buffer cache
refint
1.0
functions for implementing referential integrity (obsolete)
pg_freespacemap
1.3
examine the free space map (FSM)
insert_username
1.0
functions for tracking who changed a table
address_standardizer
3.5.4
Used to parse an address into constituent elements. Generally used to support geocoding address normalization step.
pg_visibility
1.2
examine the visibility map (VM) and page-level visibility info
pageinspect
1.13
inspect the contents of database pages at a low level
11.16.6 - PolarDB
PolarDB for PostgreSQL, with aurora flavor RAC
PolarDB is an aurora RAC flavor “cloud native” database system developed & open-sourced by Aliyun.
The latest version is v15.15.5.0, compatible with PostgreSQL 15,
and available on all linux distributions supported by Pigsty.
curl -fsSL https://repo.pigsty.io/get | bash -s v3.7.0;cd ~/pigsty;./configure -c polar # Use polar (PolarDB) template./install.yml # Run Deployment Playbook
Configure
The following parameters need to be tuned to deploy a PolarDB cluster:
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_users:- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer for meta database }pg_databases:- {name:meta ,baseline:cmdb.sql ,comment:pigsty meta database ,schemas:[pigsty]}pg_hba_rules:- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1am# PolarDB Ad Hoc Settingspg_version:15# PolarDB PG is based on PG 15pg_mode:polar # PolarDB PG Compatible modepg_packages:[polardb, pgsql-common ] # Replace PG kernel with PolarDB kernelpg_exporter_exclude_database:'template0,template1,postgres,polardb_admin'pg_default_roles:# PolarDB require replicator as superuser- {name:dbrole_readonly ,login:false ,comment:role for global read-only access }- {name:dbrole_offline ,login:false ,comment:role for restricted read-only access }- {name:dbrole_readwrite ,login:false ,roles:[dbrole_readonly] ,comment:role for global read-write access }- {name:dbrole_admin ,login:false ,roles:[pg_monitor, dbrole_readwrite] ,comment:role for object creation }- {name:postgres ,superuser:true ,comment:system superuser }- {name:replicator ,superuser:true ,replication:true ,roles:[pg_monitor, dbrole_readonly] ,comment:system replicator }# <- superuser is required for replication- {name:dbuser_dba ,superuser:true ,roles:[dbrole_admin] ,pgbouncer:true ,pool_mode:session, pool_connlimit:16 ,comment:pgsql admin user }- {name:dbuser_monitor ,roles:[pg_monitor] ,pgbouncer:true ,parameters:{log_min_duration_statement:1000 } ,pool_mode:session ,pool_connlimit:8 ,comment:pgsql monitor user }
Client Access
PolarDB for PostgreSQL is essentially equivalent to PostgreSQL 15, and any client tools compatible with the PostgreSQL wire protocol can access the PolarDB cluster.
Extensions
Most of the PGSQL module’s extension (non pure-SQL) cannot be used directly on the PolarDB kernel. If you need to use them, you need to recompile and install from source code for the new kernel.
Currently, the PolarDB kernel comes with the following 61 extension plugins. In addition to Contrib extensions, the additional extensions provided include:
name
Version
comment
adminpack
2.1
administrative functions for PostgreSQL
amcheck
1.3
functions for verifying relation integrity
autoinc
1.0
functions for autoincrementing fields
bloom
1.0
bloom access method - signature file based index
bool_plperl
1.0
transform between bool and plperl
bool_plperlu
1.0
transform between bool and plperlu
btree_gin
1.3
support for indexing common datatypes in GIN
btree_gist
1.7
support for indexing common datatypes in GiST
citext
1.6
data type for case-insensitive character strings
cube
1.5
data type for multidimensional cubes
dblink
1.2
connect to other PostgreSQL databases from within a database
dict_int
1.0
text search dictionary template for integers
dict_xsyn
1.0
text search dictionary template for extended synonym processing
earthdistance
1.1
calculate great-circle distances on the surface of the Earth
file_fdw
1.0
foreign-data wrapper for flat file access
fuzzystrmatch
1.1
determine similarities and distance between strings
hll
2.18
type for storing hyperloglog data
hstore
1.8
data type for storing sets of (key, value) pairs
hstore_plperl
1.0
transform between hstore and plperl
hstore_plperlu
1.0
transform between hstore and plperlu
hstore_plpython3u
1.0
transform between hstore and plpython3u
hypopg
1.3.1
Hypothetical indexes for PostgreSQL
insert_username
1.0
functions for tracking who changed a table
intagg
1.1
integer aggregator and enumerator (obsolete)
intarray
1.5
functions, operators, and index support for 1-D arrays of integers
isn
1.2
data types for international product numbering standards
jsonb_plperl
1.0
transform between jsonb and plperl
jsonb_plperlu
1.0
transform between jsonb and plperlu
jsonb_plpython3u
1.0
transform between jsonb and plpython3u
lo
1.1
Large Object maintenance
log_fdw
1.4
foreign-data wrapper for Postgres log file access
ltree
1.2
data type for hierarchical tree-like structures
ltree_plpython3u
1.0
transform between ltree and plpython3u
moddatetime
1.0
functions for tracking last modification time
old_snapshot
1.0
utilities in support of old_snapshot_threshold
pageinspect
1.11
inspect the contents of database pages at a low level
pase
0.0.1
ant ai similarity search
pg_bigm
1.2
text similarity measurement and index searching based on bigrams
pg_buffercache
1.4
examine the shared buffer cache
pg_freespacemap
1.2
examine the free space map (FSM)
pg_jieba
1.1.0
a parser for full-text search of Chinese
pg_prewarm
1.2
prewarm relation data
pg_repack
1.5.1-1
Reorganize tables in PostgreSQL databases with minimal locks
pg_stat_statements
1.10
track planning and execution statistics of all SQL statements executed
pg_surgery
1.0
extension to perform surgery on a damaged relation
pg_trgm
1.6
text similarity measurement and index searching based on trigrams
pg_visibility
1.2
examine the visibility map (VM) and page-level visibility info
pg_walinspect
1.0
functions to inspect contents of PostgreSQL Write-Ahead Log
pgcrypto
1.3
cryptographic functions
pgrowlocks
1.2
show row-level locking information
pgstattuple
1.5
show tuple-level statistics
plperl
1.0
PL/Perl procedural language
plperlu
1.0
PL/PerlU untrusted procedural language
plpgsql
1.0
PL/pgSQL procedural language
plpython3u
1.0
PL/Python3U untrusted procedural language
pltcl
1.0
PL/Tcl procedural language
pltclu
1.0
PL/TclU untrusted procedural language
polar_audit
1.0
provides auditing functionality
polar_feature_utils
1.0
PolarDB feature utilization
polar_io_stat
1.0
polar io stat in multi dimension
polar_login_history
1.0
record user login information
polar_masking
1.0.0
provides data masking for polardb
polar_monitor
1.0
monitor functions for PolarDB
polar_monitor_preload
1.0
examine the polardb information
polar_parameter_manager
1.1
Extension to select parameters for manger.
polar_password_policy
1.0
create password policies and check user passwords based on the policies
polar_proxy_utils
1.0
Extension to provide operations about proxy.
polar_resource_manager
1.0
a background process that forcibly frees user session process memory
polar_smgrperf
1.0
smgr perf test extension
polar_sql_mapping
1.0
Record error sqls and mapping them to correct one
polar_stat_env
1.0
env stat functions for PolarDB
polar_vfs
1.0
polar virtual file system for different storage
polar_worker
1.0
polar_worker
postgres_fdw
1.1
foreign-data wrapper for remote PostgreSQL servers
refint
1.0
functions for implementing referential integrity (obsolete)
roaringbitmap
0.5
support for Roaring Bitmaps
seg
1.4
data type for representing line segments or floating-point intervals
sslinfo
1.2
information about SSL certificates
tablefunc
1.0
functions that manipulate whole tables, including crosstab
tcn
1.0
Triggered change notifications
tsm_system_rows
1.0
TABLESAMPLE method which accepts number of rows as a limit
tsm_system_time
1.0
TABLESAMPLE method which accepts time in milliseconds as a limit
unaccent
1.1
text search dictionary that removes accents
uuid-ossp
1.1
generate universally unique identifiers (UUIDs)
vector
0.6.2
vector data type and ivfflat and hnsw access methods
xml2
1.1
XPath querying and XSLT
PolarDB for Oracle
There’s 2nd fork of PolarDB, which is PolarDB for Oracle, which is not open source.
Pigsty Pro has support for Running PolarDB for Oracle as RDS.
11.16.7 - OrioleDB
Next Gen OLTP engine for PostgreSQL
OrioleDB is a PostgreSQL storage engine extension that claims to
deliver 4x OLTP performance without the xid wraparound & table bloat, and “cloud native” (data on s3) capabilities.
You can run OrioleDB as RDS with pigsty, it is compatible with PG 17 and available on all supported Linux platforms.
The latest version is beta12 over patchset 17_11.
curl -fsSL https://repo.pigsty.io/get | bash -s v3.7.0;cd ~/pigsty;./configure -c oriole # Use the OrioleDB configuration template./install.yml # Install Pigsty with OrioleDB
For production deployments, make sure to modify the password parameters in the pigsty.yml config before running the install playbook.
Configuration
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_users:- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer for meta database }pg_databases:- {name:meta ,baseline:cmdb.sql ,comment:pigsty meta database ,schemas:[pigsty], extensions:[orioledb]}pg_hba_rules:- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1am# OrioleDB Ad Hoc Settingspg_mode:oriole # oriole compatible modepg_packages:[orioledb, pgsql-common ] # install OrioleDB kernelpg_libs:'orioledb, pg_stat_statements, auto_explain'# Load OrioleDB Extension
Usage
To use OrioleDB, you need to install the orioledb_17 and oriolepg_17 packages (currently only available as RPMs).
Initialize TPC-B-like tables with 100 warehouses using pgbench:
pgbench -is 100 meta
pgbench -nv -P1 -c10 -S -T1000 meta
pgbench -nv -P1 -c50 -S -T1000 meta
pgbench -nv -P1 -c10 -T1000 meta
pgbench -nv -P1 -c50 -T1000 meta
Next, you can rebuild these tables using the orioledb storage engine and observe the performance differences:
-- Create OrioleDB tables
CREATETABLEpgbench_accounts_o(LIKEpgbench_accountsINCLUDINGALL)USINGorioledb;CREATETABLEpgbench_branches_o(LIKEpgbench_branchesINCLUDINGALL)USINGorioledb;CREATETABLEpgbench_history_o(LIKEpgbench_historyINCLUDINGALL)USINGorioledb;CREATETABLEpgbench_tellers_o(LIKEpgbench_tellersINCLUDINGALL)USINGorioledb;-- Copy data from regular tables to OrioleDB tables
INSERTINTOpgbench_accounts_oSELECT*FROMpgbench_accounts;INSERTINTOpgbench_branches_oSELECT*FROMpgbench_branches;INSERTINTOpgbench_history_oSELECT*FROMpgbench_history;INSERTINTOpgbench_tellers_oSELECT*FROMpgbench_tellers;-- Drop original tables and rename OrioleDB tables
DROPTABLEpgbench_accounts,pgbench_branches,pgbench_history,pgbench_tellers;ALTERTABLEpgbench_accounts_oRENAMETOpgbench_accounts;ALTERTABLEpgbench_branches_oRENAMETOpgbench_branches;ALTERTABLEpgbench_history_oRENAMETOpgbench_history;ALTERTABLEpgbench_tellers_oRENAMETOpgbench_tellers;
11.16.8 - OpenHalo
MySQL Compatible Postgres 14 Fork
OpenHalo is an open-source PostgreSQL kernel that provides MySQL wire protocol compatibility.
OpenHalo is based on PostgreSQL 14.10 kernel version and provides wire protocol compatibility with MySQL 5.7.32-log / 8.0 version.
Pigsty provides deployment support for OpenHalo on all supported Linux platforms.
curl -fsSL https://repo.pigsty.io/get | bash -s v3.7.0;cd ~/pigsty;./configure -c mysql # Use MySQL (openHalo) configuration template./install.yml # Install, for production deployment please modify passwords in pigsty.yml first
For production deployment, please ensure to modify the password parameters in the pigsty.yml
configuration file before running the installation playbook.
Configuration
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_users:- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer for meta database }pg_databases:- {name:postgres, extensions:[aux_mysql]}# the mysql compatible database- {name:meta ,baseline:cmdb.sql ,comment:pigsty meta database ,schemas:[pigsty]}pg_hba_rules:- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1am# OpenHalo Ad Hoc Settingpg_mode:mysql # MySQL Compatible Mode by HaloDBpg_version:14# The current HaloDB is compatible with PG Major Version 14pg_packages:[openhalodb, pgsql-common ] # install openhalodb instead of postgresql kernel
Usage
When accessing MySQL, the actual connection uses the postgres database. Please note that the concept of “database” in MySQL actually corresponds to “Schema” in PostgreSQL. Therefore, use mysql actually uses the mysql Schema within the postgres database.
The usernames and passwords used for MySQL are the same as those in PostgreSQL. You can manage users and permissions using the standard PostgreSQL approach.
Client Access
OpenHalo provides MySQL wire protocol compatibility, listening on port 3306 by default, allowing direct connections from MySQL clients and drivers.
Pigsty’s conf/mysql configuration installs the mysql client tool by default.
You can access MySQL using the following command:
mysql -h 127.0.0.1 -u dbuser_dba
Currently, OpenHalo officially ensures that Navicat can access this MySQL port normally, but Intellij IDEA’s DataGrip access will result in errors.
Changed the default database name from halo0root back to postgres
Removed the 1.0. prefix from the default version number, reverting to 14.10
Modified the default configuration file to enable MySQL compatibility and listen on port 3306 by default
Please note that Pigsty does not provide any warranty for using the OpenHalo kernel. Any issues or requirements encountered while using this kernel should be addressed with the original vendor.
11.16.9 - Cloudberry
Cloudberry and Greenplum, the MPP data warehouse
You can deploy and monitor Cloudberry clusters, which is a Greenplum fork.
To define a Greenplum cluster, you need to specify the following parameters:
Wait for 2.0 GA
We are waiting for the official release of Apache Cloudberry 2.0, so do not use it in production now
Install
To install cloudberry, you’ll have to enable the gpsql repo module:
Set pg_mode = gpsql and the extra identity parameters pg_shard and gp_role.
#================================================================## GPSQL Clusters ##================================================================##----------------------------------## cluster: mx-mdw (gp master)#----------------------------------#mx-mdw:hosts:10.10.10.10:{pg_seq:1, pg_role:primary , nodename:mx-mdw-1 }vars:gp_role:master # this cluster is used as greenplum masterpg_shard:mx # pgsql sharding name & gpsql deployment namepg_cluster:mx-mdw # this master cluster name is mx-mdwpg_databases:- {name:matrixmgr , extensions:[{name:matrixdbts } ] }- {name:meta }pg_users:- {name:meta , password:DBUser.Meta , pgbouncer:true}- {name:dbuser_monitor , password:DBUser.Monitor , roles:[ dbrole_readonly ], superuser:true}pgbouncer_enabled:true# enable pgbouncer for greenplum masterpgbouncer_exporter_enabled:false# enable pgbouncer_exporter for greenplum masterpg_exporter_params:'host=127.0.0.1&sslmode=disable'# use 127.0.0.1 as local monitor host#----------------------------------## cluster: mx-sdw (gp master)#----------------------------------#mx-sdw:hosts:10.10.10.11:nodename:mx-sdw-1 # greenplum segment nodepg_instances:# greenplum segment instances6000:{pg_cluster:mx-seg1, pg_seq:1, pg_role:primary , pg_exporter_port:9633}6001:{pg_cluster:mx-seg2, pg_seq:2, pg_role:replica , pg_exporter_port:9634}10.10.10.12:nodename:mx-sdw-2pg_instances:6000:{pg_cluster:mx-seg2, pg_seq:1, pg_role:primary , pg_exporter_port:9633}6001:{pg_cluster:mx-seg3, pg_seq:2, pg_role:replica , pg_exporter_port:9634}10.10.10.13:nodename:mx-sdw-3pg_instances:6000:{pg_cluster:mx-seg3, pg_seq:1, pg_role:primary , pg_exporter_port:9633}6001:{pg_cluster:mx-seg1, pg_seq:2, pg_role:replica , pg_exporter_port:9634}vars:gp_role:segment # these are nodes for gp segmentspg_shard:mx # pgsql sharding name & gpsql deployment namepg_cluster:mx-sdw # these segment clusters name is mx-sdwpg_preflight_skip:true# skip preflight check (since pg_seq & pg_role & pg_cluster not exists)pg_exporter_config:pg_exporter_basic.yml # use basic config to avoid segment server crashpg_exporter_params:'options=-c%20gp_role%3Dutility&sslmode=disable'# use gp_role = utility to connect to segments
11.16.10 - Supabase
Self-hosting BaaS upon PostgreSQL
See the maintained self-hosting tutorial: Supabase
Supabase is great, but having your own Supabase is even better.
Pigsty helps you build enterprise-grade Supabase on your own servers (physical/virtual machines/cloud servers) with one-click deployment —
more extensions, better performance, deeper control, and much more cost-effective.
Pigsty is one of the three 3rd party self-hosting tutorials listed in the official Supabase docs
Quick Start
Prepare a Linux server, follow the Pigsty standard installation process, select the supabase configuration template, and execute the following commands:
Supabase is a BaaS (Backend as Service), an open-source Firebase alternative, and the most popular database + backend solution in the AI Agent era.
Supabase wraps PostgreSQL and provides authentication, messaging, edge functions, object storage, and automatically generates REST API and GraphQL API based on PostgreSQL database schemas.
Supabase aims to provide developers with a one-stop backend solution, reducing the complexity of developing and maintaining backend infrastructure.
It allows developers to eliminate most backend development work — developers only need to understand database design and frontend to quickly deliver applications!
Developers can quickly complete a full application with just frontend development and database schema design using Vibe Coding.
Currently, Supabase is the most popular open-source project in the PostgreSQL open-source ecosystem, with 80,000 stars on GitHub.
Supabase also provides “generous” free cloud service quotas for small entrepreneurs — 500 MB of free space, which is sufficient for storing user tables, view counts, and similar data.
Why Self-Host?
Since Supabase cloud service is so attractive, why self-host?
The most intuitive reason is what we mentioned in “Are Cloud Databases an Intelligence Tax?”: when your data/computing scale exceeds the cloud computing applicable spectrum (Supabase: 4C/8G/500MB free storage), costs can easily explode.
Moreover, currently, sufficiently reliable local enterprise-grade NVMe SSDs have a three to four order of magnitude advantage in cost-effectiveness compared to cloud storage, and self-hosting can better leverage this advantage.
Another important reason is functionality — Supabase cloud service functionality is limited. Many powerful PostgreSQL extensions cannot be provided as cloud services due to multi-tenant security challenges and licensing issues.
Therefore, although extensions are PostgreSQL’s core feature, only 64 extensions are available on Supabase cloud service.
Self-built Supabase with Pigsty provides up to 437 ready-to-use PostgreSQL extensions.
Additionally, autonomy and avoiding vendor lock-in are important reasons for self-hosting — although Supabase aims to provide an open-source alternative to Google Firebase without vendor lock-in, the threshold for self-building enterprise-grade Supabase to high standards is actually quite high.
Supabase includes a series of PostgreSQL extension plugins developed and maintained by them, and plans to replace the native PostgreSQL kernel with the acquired OrioleDB, but these kernels and extensions are not provided in the official PGDG repository.
This is actually a form of implicit vendor lock-in, preventing users from self-building using methods other than the supabase/postgres Docker image. Pigsty provides an open-source, transparent, and universal solution to solve this problem.
We package all 10 missing extensions developed and used by Supabase into ready-to-use RPM/DEB packages, ensuring they are available on all mainstream Linux operating system distributions:
Filter and block specific query statements using execution plan costs, C extension, provided by PIGSTY
Meanwhile, we install most extensions by default in Supabase self-hosting deployment. You can refer to the available extension list to enable them as needed.
Additionally, Pigsty handles the automatic setup of underlying high availabilityPostgreSQL database clusters, high availability MinIO object storage clusters, and even Docker container infrastructure deployment and Nginx reverse proxy, domain configuration and HTTPS certificate issuance. You can deploy any number of stateless Supabase container clusters using Docker Compose and store state in external Pigsty self-hosted database services.
In this self-hosting deployment architecture, you gain the freedom to use different kernels (PostgreSQL 15-18, OrioleDB), the freedom to install 437 extensions, the freedom to scale Supabase/Postgres/MinIO,
the freedom from database operational chores, and the freedom from vendor lock-in to run locally indefinitely. Compared to the cost of using cloud services, the price is just preparing servers and typing a few more commands.
Single Node Quick Start
Let’s start with single-node Supabase deployment. We’ll introduce multi-node high availability deployment methods later.
Before deploying Supabase, please modify the parameters (domain and passwords) in the automatically generated pigsty.yml configuration file according to your actual situation.
If it’s just local development testing, you can skip this for now. We’ll introduce how to further customize through configuration file modifications later.
If configured correctly, after about ten minutes, you can access the Supabase Studio graphical management interface locally via http://<your_ip_address>:8000.
The default username and password are: supabase and pigsty.
DockerHub blocked in mainland China
In mainland China, Pigsty uses DockerHub mirror sites provided by 1Panel and 1ms to download Supabase-related images by default, which may be slow.
You can also configure proxy and mirror sites yourself, or manually pull images with cd /opt/supabase; docker compose pull.
We also provide Supabase self-hosting expert consulting services including complete offline installation solutions.
Using Supabase object storage requires HTTPS/domain
If you need to use object storage functionality, you need to access Supabase via domain and HTTPS, otherwise errors will occur.
Please change passwords for production deployment!
For serious production deployments, must change all default passwords!
Key Technical Decisions for Self-Hosting
Here are some key technical decisions involved in self-hosting Supabase for your reference:
Using the default single-node deployment, Supabase cannot enjoy PostgreSQL/MinIO high availability capabilities.
Nevertheless, single-node deployment still has significant advantages compared to the official pure Docker Compose solution: for example, out-of-the-box monitoring systems, the ability to freely install extensions, component scaling capabilities, and providing fallback database point-in-time recovery capabilities.
If you only have one server or choose to self-host on cloud servers, Pigsty recommends using external S3 instead of local MinIO as object storage to store PostgreSQL backups and support Supabase Storage services.
Such deployment can provide a fallback-level RTO (hour-level recovery time)/RPO (MB-level data loss) disaster recovery level under single-machine deployment conditions during failures.
In serious production deployments, Pigsty recommends using at least 3-4 node deployment strategies to ensure both MinIO and PostgreSQL use multi-node deployments that meet enterprise-grade high availability requirements. In this case, you need to prepare more nodes and disks accordingly and adjust cluster configurations in the pigsty.yml configuration manifest, as well as access information in supabase cluster configuration to use high availability access points.
Some Supabase functionality requires sending emails, so SMTP services are needed. Unless purely for internal networks, for serious production deployments, using SMTP cloud services is recommended. Self-built email servers easily have their emails marked as spam and rejected.
If your service is directly exposed to the public network, we strongly recommend using real domains and HTTPS certificates and accessing through Nginx Portal.
Next, we’ll discuss some advanced topics in sequence: how to further improve Supabase security, availability, and performance based on single-node deployment.
Advanced Topic: Security Hardening
Pigsty Base Components
For serious production deployments, we strongly recommend changing Pigsty default passwords.
Because these default values are public and well-known, going to production without changing passwords is like streaking:
Generate a JWT_SECRET longer than 40 characters and use the tools in the tutorial to sign ANON_KEY and SERVICE_ROLE_KEY JWTs.
Use the tools provided in the tutorial to generate an ANON_KEY JWT based on JWT_SECRET and expiration time attributes. This is the credential for anonymous users.
Use the tools provided in the tutorial to generate a SERVICE_ROLE_KEY based on JWT_SECRET and expiration time attributes. This is the credential for higher-privilege service roles.
Setup PG_META_CRYPTO_KEY with a random string at least 32 char long for securing connection strings between Studio and postgres-meta
If your PostgreSQL business user uses a password different from the default, please modify the POSTGRES_PASSWORD value accordingly
If your object storage uses a password different from the default, please modify the S3_ACCESS_KEY and S3_SECRET_KEY values accordingly
After modifying Supabase credentials, you can restart Docker Compose containers to apply the new configuration:
./app.yml -t app_config,app_launch
cd /opt/supabase; make up
Advanced Topic: Domain Integration
If you’re using Supabase on localhost or within a LAN, you can choose IP:Port direct connection to Kong’s exposed HTTP port 8000 to access Supabase.
You can use an internal static DNS domain, but for serious production deployments, we recommend using real domain + HTTPS to access Supabase.
In this case, your server should have a public IP address, you should own a domain, use DNS resolution services provided by cloud/DNS/CDN providers to point it to the installation node’s public IP (optional fallback: local /etc/hosts static resolution).
A simple approach is to batch replace the placeholder domain (supa.pigsty) with your actual domain, say supa.pigsty.cc:
sed -ie 's/supa.pigsty/supa.pigsty/g' ~/pigsty/pigsty.yml
If you haven’t configured it beforehand, reload Nginx and Supabase configurations:
make nginx # Reload nginx configurationmake cert # Apply for free HTTPS certificate with certbot./app.yml # Reload Supabase configuration
The modified configuration should look like the following snippet:
all:vars:infra_portal:supa :domain:supa.pigsty.cc # Replace with your domain!endpoint:"10.10.10.10:8000"websocket:truecertbot:supa.pigsty.cc # Certificate name, usually same as domainchildren:supabase:vars:supabase:# the definition of supabase appconf:# override /opt/supabase/.envSITE_URL:https://supa.pigsty # <------- Change This to your external domain nameAPI_EXTERNAL_URL:https://supa.pigsty # <------- Otherwise the storage api may not work!SUPABASE_PUBLIC_URL:https://supa.pigsty # <------- DO NOT FORGET TO PUT IT IN infra_portal!
Complete domain/HTTPS configuration can refer to the Certificate Management tutorial. You can also use Pigsty’s built-in local static resolution and self-signed HTTPS certificates as fallback.
Advanced Topic: External Object Storage
You can use S3 or S3-compatible services as object storage for PostgreSQL backups and Supabase usage. Here we use Alibaba Cloud OSS object storage as an example.
Pigsty provides a terraform/spec/aliyun-meta-s3.tf template
that can be used to deploy a server and an OSS bucket on Alibaba Cloud.
First, modify the S3-related configuration in all.children.supa.vars.apps.[supabase].conf, pointing it to the Alibaba Cloud OSS bucket:
# if using s3/minio as file storageS3_BUCKET:data # Replace with S3-compatible service connection informationS3_ENDPOINT:https://sss.pigsty:9000 # Replace with S3-compatible service connection informationS3_ACCESS_KEY:s3user_data # Replace with S3-compatible service connection informationS3_SECRET_KEY:S3User.Data # Replace with S3-compatible service connection informationS3_FORCE_PATH_STYLE:true# Replace with S3-compatible service connection informationS3_REGION:stub # Replace with S3-compatible service connection informationS3_PROTOCOL:https # Replace with S3-compatible service connection information
Reload Supabase configuration with the following command:
./app.yml -t app_config,app_launch
You can also use S3 as PostgreSQL backup repository by adding an aliyun backup repository definition in all.vars.pgbackrest_repo:
all:vars:pgbackrest_method:aliyun # pgbackrest backup method: local,minio,[other user-defined repositories...], in this example backup is stored to MinIOpgbackrest_repo:# pgbackrest backup repository: https://pgbackrest.org/configuration.html#section-repositoryaliyun:# Define a new backup repository aliyuntype:s3 # Alibaba Cloud OSS is S3-compatible object storages3_endpoint:oss-cn-beijing-internal.aliyuncs.coms3_region:oss-cn-beijings3_bucket:pigsty-osss3_key:xxxxxxxxxxxxxxs3_key_secret:xxxxxxxxs3_uri_style:hostpath:/pgbackrestbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest.MyPass # Set an encryption password, pgBackrest backup repository encryption passwordretention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for the last 14 days
Then specify using the aliyun backup repository in all.vars.pgbackrest_method and reset pgBackrest backup:
./pgsql.yml -t pgbackrest
Pigsty will switch the backup repository to external object storage. More backup configurations can refer to PostgreSQL Backup documentation.
Advanced Topic: Using SMTP
You can use SMTP to send emails by modifying the supabase application configuration and adding SMTP information:
all:children:supabase:# supa groupvars:# supa group varsapps:# supa group app listsupabase:# the supabase appconf:# the supabase app conf entriesSMTP_HOST:smtpdm.aliyun.com:80SMTP_PORT:80SMTP_USER:[email protected]SMTP_PASS:your_email_user_passwordSMTP_SENDER_NAME:MySupabaseSMTP_ADMIN_EMAIL:[email protected]ENABLE_ANONYMOUS_USERS:false
Don’t forget to use app.yml to reload the configuration
Advanced Topic: True High Availability
After these configurations, you have an enterprise-grade Supabase (basic single-machine version) with public domain, HTTPS certificate, SMTP, PITR backup, monitoring, IaC, and 400+ extensions.
For high availability configuration, please refer to other parts of Pigsty documentation. If you’re too lazy to read and learn, we provide hands-on Supabase self-hosting expert consulting services — ¥2000 to save you from the hassle of tinkering and downloading.
Single-node RTO/RPO relies on external object storage services for fallback. If your node fails, backups are retained in external S3 storage, and you can redeploy Supabase on a new node and restore from backup.
Such deployment can provide a minimum standard RTO (hour-level recovery time)/RPO (MB-level data loss) fallback disaster recovery level during failures.
To achieve RTO < 30s with zero data loss failover, you need to use multi-node high availability deployment, which involves:
ETCD: DCS needs three or more nodes to tolerate one node failure.
PGSQL: PostgreSQL synchronous commit mode without data loss, recommend using at least three nodes.
INFRA: Monitoring infrastructure failure has less impact, recommend using dual replicas in production
Supabase stateless containers themselves can also be multi-node replicas to achieve high availability.
In this case, you also need to modify PostgreSQL and MinIO access points to use DNS/L2 VIP/HAProxy and other high availability access points
For these parts, you only need to refer to the documentation of each module in Pigsty for configuration and deployment.
We recommend referring to the configurations in conf/ha/trio.yml and conf/ha/safe.yml to upgrade cluster scale to three nodes or more.
11.16.11 - FerretDB
Mongo Wire Compatible PostgreSQL
FerretDB is an open-source MongoDB wire protocol compatible middleware that allows you to use PostgreSQL as a drop-in replacement for MongoDB.
It enables applications that rely on MongoDB’s wire protocol to work seamlessly with PostgreSQL, providing a bridge between the two databases.
To enable FerretDB, you’ll need the FerretDB patched documentdb extension, which is also available in the Pigsty repository.
The latest combo is FerretDB 2.7 and DocumentDB 0.107.0.
./configure -c mongo # Use FerretDB / DocumentDB config template./install.yml # Install, for production deployment please modify passwords in pigsty.yml first
For production deployment, please ensure to modify the password parameters in the pigsty.yml configuration file before running the installation playbook.
Configuration
pg-meta:hosts:10.10.10.10:{pg_seq:1, pg_role:primary }vars:pg_cluster:pg-metapg_users:- {name:mongod ,password:DBUser.Mongo ,pgbouncer:true ,roles:[dbrole_admin ] ,comment:ferretdb super user ,superuser:true}- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin ] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer }pg_databases:- {name:meta, owner:mongod ,baseline:cmdb.sql ,comment:pigsty meta database ,schemas:[pigsty] ,extensions:[documentdb, postgis, vector, pg_cron, rum ]}pg_hba_rules:- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}- {user:mongod , db:all ,addr:world ,auth:pwd ,title:'mongodb password access from everywhere'}node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1am# DocumentDB Settingspg_extensions:[documentdb, citus, postgis, pgvector, pg_cron, rum ]pg_libs:'pg_documentdb, pg_documentdb_core, pg_cron, pg_stat_statements, auto_explain'# add timescaledb to shared_preload_librariespg_parameters:{cron.database_name:meta }
You can access FerretDB using MongoDB connection strings with any MongoDB driver in any language. Here’s an example using the mongosh CLI tool:
$ mongosh
Current Mongosh Log ID: 67ba8c1fe551f042bf51e943
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.4.0
Using MongoDB: 7.0.77
Using Mongosh: 2.4.0
For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/
test>
mongosh 'mongodb://dbuser_meta:[email protected]:27017/meta'# Business admin usermongosh 'mongodb://dbuser_view:[email protected]:27017/meta'# Read-only user
Quick Start
You can connect to FerretDB and use it as if it were a MongoDB cluster.
Harness the synergistic power of PostgreSQL extensions
Pigsty allows you to harness the synergistic superpower of the Postgres extensions ecosystem with 3 things: Catalog, Repo, and pig.
Extension Catalog
The complete list of <span class="text-lg font-black text-emerald-500">437</span> available PostgreSQL extensions
Software Repository
The APT/YUM repo that deliver PostgreSQL extensions
Package Manger
The missing package manager for PostgreSQL & Extensions
Quick Start
How to get, install, config, manage these extensions?
There are 437 PostgreSQL extensions in the v3.7.0 catalog. PostgreSQL 18 is the default in v3.7.0.
The per-major figures below are the archive’s PG13–17 compatibility snapshot; the final PG18 breakdown was not recorded in this table. Use the v3.7.0 package aliases and release note for PG18.
[**Download**](#download-extension) : Which extension packages to download
```yaml tab="config" title="define which extensions to be downloaded"
repo_extra_packages: [ postgis, timescaledb, vector ]
```
```bash tab="apply" title="download package"
make repo
```
Step 2
[Install](#install-extension) : Which extensions to be installed
```yaml tab="config"
pg_extensions: [ postgis, pgvector, timescaledb ]
```
```bash tab="apply"
./pgsql.yml -t pg_ext # install extensions
```
Step 3
[**Load**](#load-extension) : Which extensions to be pre-loaded
```yaml tab="config"
pg_libs: 'timescaledb, pg_stat_statements, auto_explain' # add extension to preload libraries (not all extensions need this)
```
```bash tab="apply" title="edit existing cluster config and reload"
pg edit-config --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'
```
Step 4
[**Create**](#create-extension) : Create extension in the [database](/docs/pgsql/db)
```yaml tab="config"
pg_databases:
- { name: meta ,extensions: [ postgis, timescaledb, vector ] }
```
```sql tab="apply" title="create extension in existing database"
CREATE EXTENSION postgis CASCADE;
```
Quick Start
You can describe extensions in the config inventory, and pigsty will
download, install, configure, and enable extensions for you.
This example makes postgis, pgvector, timescaledb out-of-the-box:
To install the right RPM / DEB with minimal effort, we need to use the abstract layer: package alias.
So you can install these extensions by specifying the “Normalized” names, like pgvector or postgis.
Without knowing any details about PG & OS version, Arch, Extension versions, and any other details.
Package alias pkg are used for extension download & install, but you’ll have to use the extension name ext when CREATE EXTENSION in the database (like the vector in meta database).
And beware some extensions require explicit preloading, like the timescaledb in the above example.
Besides, all the extensions are categorized into 16 major categories, we also have alias for the entire extension category
so that you can download and install them in batch, such as:
All extensions CAN be installed simultaneously, except the olap category, where citus conflict with hydra, and pg_duckdb conflict with pg_mooncake.
So you can download them all, but install one at a time.
11.17.3 - Download
Download PostgreSQL Extension
In Pigsty, downloading and installing extensions are separate steps. During INFRA module installation, Pigsty downloads all required software to the local machine and creates a local YUM/APT repo for the entire deployment.
This approach accelerates installation, eliminates redundant downloads, removes the need for database nodes to access the internet, reduces network traffic, improves delivery reliability, and ensures consistent versions across your environment - all best practices for production deployments.
For development environments, installing extensions directly from internet repo is also acceptable
Quick Start
Packages defined in repo_packages and repo_extra_packages are automatically downloaded to your local repo during Pigsty installation.
For PostgreSQL-related packages (core and extensions), typically put them in repo_extra_packages while leaving repo_packages with its os-specific global defaults.
The default value for repo_extra_packages is [pgsql-main], an alias representing core PostgreSQL and critical extensions for the current active major version.
repo_extra_packages:[pgsql-main ] # main packages (kernel + 3 extension) for current pg major 18
To add specific extensions, simply add Pigsty extension package name (pkg) to this parameter. Pigsty automatically downloads the appropriate packages for your active PG version and current OS distro.
repo_extra_packages:[pgsql-main, documentdb, citus, postgis, pgvector, pg_cron, rum ]
To download all available extensions for the current PG version, add all 16 extension category aliases (as in the rich config template):
PostgreSQL has a rich open-source ecosystem with numerous packages across different systems and architectures.
Pigsty provides an abstraction layer that categorizes PostgreSQL packages into “aliases,” hiding differences between systems, architectures, and PG versions.
In the Quick Start section, we used aliases like pgsql-main and pgsql-core. These aliases are translated into specific package names based on your system and architecture. For EL systems, pgsql-main expands to postgresql$v* kernel packages with pgvector_$v*, pg_repack_$v*, and wal2json_$v* extension packages.
The $v placeholder is replaced by the pg_version value (default: 18) to target the correct version. The * wildcard expands to include all package variants (e.g., server, libs, contrib, devel). Pigsty handles these details automatically.
The complete list of available packages and aliases is in roles/node_id/vars/<os_package>.yml. Here are commonly used aliases available across all supported systems:
Install all extensions simultaneously is applicable (except two conflicts in the olap category) but not recommended. Just install the extensions you need by explicitly specifying them in the pg_extensions parameter.
Configure
During PGSQL cluster init, Pigsty will automatically install packages (& alias) specified in pg_packages and pg_extensions.
Both parameters can be used to install PostgreSQL-related packages. Typically, pg_packages is used to globally specify packages that should be installed across all PostgreSQL clusters in your environment: such as the PostgreSQL kernel, high-availability agent like Patroni, connection pooling with pgBouncer, monitoring with pgExporter, etc.
By default, Pigsty also specifies 3 important extensions here: pgvector, pg_repack, and wal2json for vector search, bloat management, and CDC change extraction.
Meanwhile, pg_extensions is usually used to specify extension for a specific cluster. The default is an empty list, indicating no other extensions will be installed by default.
pg_packages:# pg packages to be installed, alias can be used, state=present- postgresql- wal2json pg_repack pgvector- patroni pgbouncer pgbackrest pg_exporter pgbadger vip-managerpg_extensions:[]# pg extensions to be installed, alias can be used, state=latest
An important distinction: packages installed via pg_packages are merely ensured to be present, whereas those installed via pg_extensions are automatically upgraded to the latest available version.
When using a local software repo, this distinction isn’t an issue. However, when using upstream internet repo, consider this carefully and move extensions you don’t want automatically upgraded to pg_packages.
Install
Extensions pre-defined in the pg_extensions (and pg_packages) will be installed during cluster provisioning.
To install new extensions on a provisioned PostgreSQL cluster:
First, add extensions to pg_extensions, then execute the playbook subtask:
./pgsql.yml -t pg_extension # install extensions specified in pg_extensions
Note that extension plugins specified in the pg_extension task will be upgraded to the latest available version in your current environment by default.
Repo
To install extension, you need to ensure one of the following conditions is met:
Local Repo: You have configured using Pigsty’s local repo, and the extensions have already been downloaded to the local repo.
Online Repo: You have directly configured upstream internet repo on the target node, and internet access is available on these nodes.
For production environments, we recommend using Pigsty’s local software repo to manage and install extensions uniformly: First download extensions to the local repo, then install them from there.
This ensures consistent extension versions across your environment and prevents database nodes from directly accessing the internet. You have to do nothing when installed from local repo, just make sure they are downloaded to the local repo.
For development environments, you may choose to directly use upstream internet repo for convenience. Use the following commands to add Internet repo and install extensions on the target cluster directly:
./node.yml -l <cls> -t node_repo -e node_repo_modules=local,node,pgsql # Enable internet repo on target node./pgsql.yml -l <cls> -t pg_extension # Install extensions using local+internet upstream repos
Package Alias
When installing extensions, users can use extension aliases to specify extension.
The aliases will be translated to the current active PG major version and OS environment.
pgaudit got a different naming pattern on el for pg 15-: pg16+ = pgaudit, pg15=pgaudit17, pg14=pgaudit16 pg13=pgaudit15 pg12=pgaudit14
postgis got its own version in el package name: postgis35 by default, and postgis33 for legacy el7
11.17.5 - Config
Preload extensions and configure extension parameters
While most PostgreSQL extensions written in SQL can be directly enabled with CREATE EXTENSION, some extensions that use special postgres hook will require an extra step to preload them before using.
Preloading
Most extensions have one or more corresponding dynamic library (.so, .dylib, .dll), some of them require preloading before using.
Attempting to CREATE these extensions without proper preloading will result in an error.
And a wrongly configured preload library may lead to a failure on database restart/start.
Some extensions can partially work without preloading, which means part of the extension features are available directly, and the rest of the features are available after preloading.
To preload an extension, add it to the shared_preload_libraries and restart the database server.
The Extension Catalog gives the complete list of extensions that require dynamic preloading.
Configure
To configure a preload on new postgres cluster, the pg_libs parameter can be used.
It will be populated to the shared_preload_libraries parameter during postgres cluster bootstrap.
Example: Setup Supabase Extension Preloading
This example show how to specify pre-loaded extensions with pg_libs parameter.
shared_preload_libraries is a comma-separated list of extensions.
Beware that only works before cluster creation. After that,
you’ll have to config cluster to change the shared_preload_libraries parameter on existing cluster. (with patronictl, ALTER SYSTEM, etc…)
If you want to configure preloading manually, you can just change the postgresql.conf by yourself
Default
The default value of pg_libs is pg_stat_statements, auto_explain,
which preload these two Contrib extensions by default, these two extensions provide essential observability:
auto_explain: Automatic logging of slow query execution plans
pg_stat_statements: Tracks planning and execution statistics for grouped SQL statements
Caveats
Preload libraries are loaded one by one, so the order of extensions in shared_preload_libraries matters,
Here are some known rules to follow:
For STAT extension, add them AFTERpg_stat_statements to ensure using the same query_id.
timescaledb and citus should be placed at the BEGINNING of shared_preload_libraries
If you use citus and timescaledb together, place citus before timescaledb.
Use pg_documentdb and pg_documentdb_core as library name for documentdb.
pg_search does not require preloading in PostgreSQL 17 and later, but earlier versions do.
Parameter
Some extensions have configurable parameters, you can manage them in different places.
pg_parameters: write to /pg/data/postgresql.auto.conf
Extensions need to be installed first, some extension also requires preloading before using.
Some extensions have dependencies on other extensions.
In such cases, you can either install the dependencies first
or use the CASCADE clause to install all dependencies at once.
CREATEEXTENSIONdocumentdbCASCADE;-- create documentdb extension and all its dependencies
You can also provision extension with Pigsty, which will automatically create the extensions for you.
Configure
Extensions (database logical objects) are logically part of PostgreSQL databases.
In Pigsty, you can specify which extensions to be created in a database with pg_databases parameter.
But you can explicitly specify extension details with the object format, like create them in a specific schema.
Or install a specific version. Here’s a complete example (self-hosting supabase):
pg_databases:- name:postgresbaseline:supabase.sqlschemas:[extensions ,auth ,realtime ,storage ,graphql_public ,supabase_functions ,_analytics ,_realtime ]extensions:# Extensions to be enabled in the postgres database- {name:pgcrypto ,schema:extensions }# cryptographic functions- {name:pg_net ,schema:extensions }# async HTTP- {name:pgjwt ,schema:extensions }# json web token API for postgres- {name:uuid-ossp ,schema:extensions }# generate universally unique identifiers (UUIDs)- {name:pgsodium } # pgsodium is a modern cryptography library for Postgres.- {name:supabase_vault } # Supabase Vault Extension- {name:pg_graphql } # pg_graphql: GraphQL support- {name:pg_jsonschema } # pg_jsonschema: Validate json schema- {name:wrappers } # wrappers: FDW collections- {name:http } # http: allows web page retrieval inside the database.- {name:pg_cron } # pg_cron: Job scheduler for PostgreSQL- {name:timescaledb } # timescaledb: Enables scalable inserts and complex queries for time-series data- {name:pg_tle } # pg_tle: Trusted Language Extensions for PostgreSQL- {name:vector } # pgvector: the vector similarity search- {name:pgmq } # pgmq: A lightweight message queue like AWS SQS and RSMQ
Define Extension
The extensions field is a list of extension (name or object) to be created in the database.
It will be created under the first schema in dbsu’s search_path, (usually the public schema).
Here, the extensions in the database object is a list where each element can be:
A simple string representing the extension name, such as vector
Alternatively, A dictionary that may contain the following fields can be used:
schema: Schema for installing the extension, OPTIONAL, defaults to the first schema in the current dbsu search path, usually the default public.
version: Specifies the extension version, OPTIONAL, defaults to the latest version, rarely used.
If the database doesn’t exist yet, the extensions defined here will be automatically created when creating a cluster or creating a database through Pigsty.
Re-creating database with non-trivial baseline schema may be dangerous (if you put some DROP there)
So for existing clusters / databases, it’s advised to use your own schema migration tool to manage extensions. (pgadmin, psql, bytebase, flyway, sqlitch,…)
But it’s helpful to enlist them in the config inventory for bookkeeping purposes. (So if you want to fork this cluster, it includes these extensions)
Default Extension
Some built-in extensions and one special pg_repack are created by default in Pigsty.
These extensions are defined by pg_default_extensions, created in the template1 database and the postgres database by default.
Newly created databases will inherit these extensions from template1, so you don’t need to create them again.
One extra default schema monitor is defined by pg_default_schemas is also created by default.
Which is used to contain monitoring related extensions, tables, functions and views.
There are three 3rd-party extensions that are available by default in Pigsty:
The pg_repack extension is an important utility for maintaining bloat tables online.
vector is a very popular extension for RAG,
It is installed by default (in the pgsql-mainalias) and created in the placeholder meta database in most config template.
The wal2json is another important extension for Changing Data Capture (CDC). It is installed by default, but it is an extension without DDL,
So you don’t need to CREATE it explicitly.
PostgreSQL extensions typically consist of three parts: a required control file, optional SQL files, and optional libraries.
If an extension does not have SQL file, CREATE EXTENSION command is not needed.
How to update PostgreSQL extensions to newer versions
To update an existing extension, you need to first update the RPM/DEB package with your OS’s package manager,
then alter the extension to the new version in PostgreSQL with ALTER EXTENSION ... UPDATE.
You can upgrade extension packages with the following commands
pig ext update extname...
yum upgrade extname...
apt upgrade extname...
./pgsql.yml -t pg_ext # -l cls
All extensions listed in pg_extensions will be upgraded using during the pgsql.yml playbook execution.
This playbook will automatically install the latest available version of extension RPM/DEB packages in your current environment.
(from built local repo or via Internet directly).
You can also upgrade extensions with linux system’s yum/apt upgrade command directly, but you need to specify the full package names:
yum upgrade extname...
apt upgrade extname...
Pigsty’s pig cli can also help you with that, without the burden of specifying full package names:
If the TO new_version clause is omitted, the extension will be updated to the latest version available.
11.17.8 - Remove
How to remove PostgreSQL extensions
Remove Extension
To uninstall an extension, you typically need to run the DROP EXTENSION SQL statement:
DROPEXTENSION"<extname>";
If other extensions or database objects depend on this extension, you’ll need to remove those dependencies first before uninstalling the extension.
Or remove all of them with CASCADE option:
DROPEXTENSION"<extname>"CASCADE;
Warning
The CASCADE option will delete all objects that depend on this extension,
including database objects, functions, views, etc. Use with caution!
Some extensions don’t have DDL, these extensions do not require the DROP EXTENSION statement to uninstall.
Instead, you can simply remove the extension from the shared_preload_libraries (if configured) and uninstall the package.
Refer to the Extensions Without DDL section for more details.
Remove Loading
If you’re using an extension that requires dynamic loading (which modifies the shared_preload_libraries parameter), you need to first re-confnigure the shared_preload_libraries parameter.
Remove the extension name from shared_preload_libraries and restart the database cluster for the changes to take effect.
After removing the extension (logical object) from all databases in the cluster, you can safely uninstall the extension’s software package. Ansible commands can help you do this conveniently:
ansible <cls> -m package -a "name=<extname> state=absent"
You can also use pig, or apt/yum commands directly to uninstall.
If you don’t know the extension package name, you can refer to the Extension List or check the extension package name mapping defined in roles/node_id/vars.
12 - Infra
The portal and observability stack
Architecture
Architecture, core concepts, identity management
Configuration
Configure infra module, and use multiple infra nodes.
Parameters
Customize infra components with 57+ parameters
Administration
Manage local repo, nginx portal, domain, certs, and more
Playbooks
Ansible playbooks that can be used in this module
Monitoring
Dashboards, metrics, record & alerting rules.
FAQ
Frequently Asked Questions about infra module
12.1 - Architecture
Architecture and entities in the infra module
A standard Pigsty deployment comes with an INFRA module that provides the following services:
These are essential for a production-grade PostgreSQL service, and will be installed by default.
Component
Port
Domain
Description
nginx
80
h.pigsty
Web Service Portal (YUM/APT Repo)
alertmanager
9059
a.pigsty
Alert Aggregation and delivery
prometheus
9058
p.pigsty
Monitoring Time Series Database
grafana
3000
g.pigsty
Visualization Platform
loki
3100
-
Logging Collection Server
pushgateway
9091
-
Collect One-Time Job Metrics
blackbox_exporter
9115
-
ICMP, TCP, HTTP Probing
dnsmasq
53
-
DNS Server, optional
chronyd
123
-
NTP Time Server, optional
HA PG can be deployed without INFRA
If you don’t want these, the Minimal Install mode deploy HA Postgres without the Infra module.
Nginx: Acts as a web server for local repo and a reverse proxy for other web UI services
Grafana : Visualization platform for metrics, dashboards, and data analytics
Loki: Centralized log aggregation and querying via Grafana
Prometheus: Time-series monitoring database for metrics collection, storage, and alert evaluation
AlertManager: Alert aggregation, notification dispatch, and silencing
PushGateway: Collects metrics from one-off and batch jobs
DNSMASQ: DNS resolution for Pigsty’s internal domains
Chronyd: NTP time synchronization to keep all nodes in sync
The INFRA module isn’t mandatory for HA PostgreSQL - for instance, it’s omitted in Slim Install mode.
However, since INFRA provides essential supporting services for production-grade HA PostgreSQL clusters, it’s strongly recommended for most deployments.
If you already have your own infrastructure (Nginx, local repos, monitoring, DNS, NTP), you can disable INFRA and configure Pigsty to use your existing stack instead.
Nginx
Nginx is the gateway for all WebUI services in Pigsty, serving on HTTP (80) / HTTPS (443) by default.
It exposes web UIs like Grafana, Prometheus, AlertManager, and HAProxy console, while also serving static resources like local yum/apt repos.
Nginx configuration follows infra_portal definitions, for example:
These endpoint definitions are referenced by other services - logs go to loki endpoint, Grafana datasources register to grafana endpoint, alerts route to alertmanager endpoint.
Pigsty allows rich Nginx customization as a local file server or reverse proxy, with self-signed or real HTTPS certs.
During installation, Pigsty creates a local software repository on the INFRA node to speed up subsequent software installations.
Located at /www/pigsty and served by Nginx, it’s accessible via http://h.pigsty/pigsty.
Pigsty’s offline package is a tarball of a pre-built repo directory. If /www/pigsty exists with a /www/pigsty/repo_complete marker, Pigsty skips downloading from upstream - perfect for air-gapped environments!
Repo definition lives in /www/pigsty.repo, fetchable via http://${admin_ip}/pigsty.repo:
Prometheus, our TSDB for monitoring, listens on port 9058 (access via IP:9058 or http://p.pigsty).
Key features:
Service discovery via local static files with identity info
Metric scraping, pre-processing, and TSDB storage
Alert rule evaluation and forwarding to AlertManager
AlertManager
AlertManager handles alerts on port 9059 (IP:9059 or http://a.pigsty). While it receives Prometheus alerts, you’ll need extra config (e.g., SMTP) for notifications.
Configs for Prometheus, AlertManager, PushGateway, and BlackboxExporter are in: Config: INFRA - PROMETHEUS
Grafana
Grafana, our visualization powerhouse, runs on port 3000 (IP:3000 or http://g.pigsty).
Pigsty’s monitoring is dashboard-based with URL-driven navigation. Drill down or up quickly to pinpoint issues.
Fun fact: We’ve supercharged Grafana with extra viz plugins like ECharts - it’s not just monitoring, it’s a low-code data app platform!
Loki handles logs on port 3100, with Promtail shipping logs from nodes to the mothership.
Ansible is already installed on the Admin Node during bootstrap.
So the ansible on infra nodes are not actually used.
But it gives you a viable backup option in case your admin node is compromised.
DNSMASQ
DNSMASQ handles DNS resolution, with other modules registering their domains to INFRA’s DNSMASQ service.
DNS records live in /etc/hosts.d/ on all INFRA nodes.
This is purely optional, if you already have your own NTP servers configured, just leave it alone.
12.2 - Configure
configure infra nodes, nginx, repo, dns, ntp, monitor system
The INFRA module primarily provides monitoring infrastructure and is optional for PostgreSQL service.
Unless you have manually configured dependencies on DNS/NTP services from INFRA nodes elsewhere, failures in the INFRA module typically won’t affect the normal operation of PostgreSQL database clusters.
In most cases, a single INFRA node is sufficient for typical scenarios. For production environments with higher requirements, we recommend using 2-3 INFRA nodes for high availability.
To improve resource utilization, PostgreSQL high availability typically relies on the ETCD module, which can share nodes with the INFRA module.
Using more than 3 INFRA nodes provides limited benefits, but you can use more ETCD nodes (e.g., 5) to enhance the availability and reliability of DCS services.
Examples
To install the INFRA module on nodes, first add node IPs to the infra group in the inventory and assign them an Infra instance number infra_seq.
By default, a single INFRA node configuration meets most requirements. All configuration templates include a default infra group definition:
The 10.10.10.10 IP placeholder in the infra group will be replaced with the current node’s primary IP address during configuration, meaning the INFRA module will be installed on the current node.
Then use the infra.yml playbook to initialize the INFRA module on the node.
More Nodes
To configure two INFRA nodes, add new IPs to infra.hosts:
Most components in the Infra module are “stateless/shared-state”. For these components, high availability primarily requires addressing load balancing.
Infra component load balancing can be achieved through two methods: Keepalived L2 VIP or HAProxy Layer 4 Load Balancing.
If your network environment supports Layer 2 connectivity, you can use Keepalived L2 VIP for high availability:
This section contains metadata for current Pigsty deployments, such as version string, admin node IP address, repository mirror region and HTTP(S) proxy when downloading packages.
version:v3.7.0 # Pigsty version stringadmin_ip:10.10.10.10# admin node ip addressregion:default # upstream mirror region: default,china,europeproxy_env:# global proxy env when downloading packagesno_proxy:"localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"# http_proxy: # set your proxy here: e.g http://user:[email protected]# https_proxy: # set your proxy here: e.g http://user:[email protected]# all_proxy: # set your proxy here: e.g http://user:[email protected]
version
type: string, level: G
pigsty version string
default value:v3.7.0
It will be used for pigsty introspection & content rendering.
admin_ip
type: ip, level: G
admin node ip address
default value:10.10.10.10
Node with this ip address will be treated as admin node, usually point to the first node that install Pigsty.
The default value 10.10.10.10 is a placeholder that will be replaced during configure
This parameter is referenced by many other parameters, such as:
The exact string ${admin_ip} will be replaced with the actual admin_ip for above parameters.
region
type: enum, level: G
upstream mirror region: default,china,europe
default value: default
If a region other than default is set, and there’s a corresponding entry in repo_upstream.[repo].baseurl, it will be used instead of default.
For example, if china is used, pigsty will use China mirrors designated in repo_upstream if applicable.
proxy_env
type: dict, level: G
global proxy env when downloading packages
default value:
proxy_env:# global proxy env when downloading packageshttp_proxy:'http://username:[email protected]'https_proxy:'http://username:[email protected]'all_proxy:'http://username:[email protected]'no_proxy:"localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.aliyuncs.com,mirrors.tuna.tsinghua.edu.cn,mirrors.zju.edu.cn"
It’s quite important to use an HTTP proxy in restricted production environments, or when your Internet access is blocked (e.g., Mainland China).
Please note that if the Docker module is used, the proxy server configuration will also be written to the Docker Daemon configuration file.
Please note that if the -x parameter is specified during ./configure, the proxy configuration information in the current environment will be automatically filled into the generated pigsty.yaml file.
CA
Self-Signed CA used by Pigsty. It is required to support advanced security features.
ca_create:true# create CA if missing; otherwise require supplied key/certca_cn:pigsty-ca # CA common name, fixed as pigsty-cacert_validity:7300d # cert validity, 20 years by default
ca_create
type: bool, level: G
Default value: true. The ca role creates files/pki/ca/ca.key and
files/pki/ca/ca.crt only when they are missing. Existing CA material is
always reused. Set this to false when supplying an existing key and
certificate; the role aborts if the key is then missing.
ca_cn
type: string, level: G
CA common name, not recommended to change it.
default value: pigsty-ca
You can check that with openssl x509 -text -in /etc/pki/ca.crt.
cert_validity
type: interval, level: G
certificate validity, 20 years by default, which is sufficient for most scenarios
default value: 7300d
INFRA_ID
Infrastructure identity and portal definition.
#infra_seq: 1 # infra node identity, explicitly requiredinfra_portal:# infra services exposed via portalhome :{domain:h.pigsty }grafana :{domain:g.pigsty ,endpoint:"${admin_ip}:3000",websocket:true}prometheus :{domain:p.pigsty ,endpoint:"${admin_ip}:9058"}alertmanager :{domain:a.pigsty ,endpoint:"${admin_ip}:9059"}blackbox :{endpoint:"${admin_ip}:9115"}loki :{endpoint:"${admin_ip}:3100"}
infra_seq
type: int, level: I
infra node identity, REQUIRED, no default value, you have to assign it explicitly.
infra_portal
type: dict, level: G
infra services exposed via portal.
default value will expose home, grafana, prometheus, alertmanager via nginx with corresponding domain names.
infra_portal:# infra services exposed via portalhome :{domain:h.pigsty }grafana :{domain:g.pigsty ,endpoint:"${admin_ip}:3000",websocket:true}prometheus :{domain:p.pigsty ,endpoint:"${admin_ip}:9058"}alertmanager :{domain:a.pigsty ,endpoint:"${admin_ip}:9059"}blackbox :{endpoint:"${admin_ip}:9115"}loki :{endpoint:"${admin_ip}:3100"}
Each record consists of a Key and a Value dictionary, where name is the key, representing the component name, and the value is an object that can configure the following parameters:
Each record consists of a Key and a Value dictionary, where name is the key, representing the component name, and the value is an object that can configure the following parameters:
Each record consists of a Key and a Value dictionary, where name is the key, representing the component name, and the value is an object that can configure the following parameters:
name: REQUIRED, specify the name of the Nginx server
Default records: home, grafana, prometheus, alertmanager, blackbox, loki are fixed names, please do not modify them.
Used as part of the Nginx configuration file name, corresponding to the configuration file: /etc/nginx/conf.d/<name>.conf
Nginx servers without the domain field will not generate configuration files, but will be used as references.
domain: OPTIONAL, when the service needs to be exposed to the outside world via Nginx, it is a REQUIRED field, specifying the domain name used
In Pigsty self-signed Nginx HTTPS certificate, the domain name will be added to the SAN field of the Nginx SSL certificate
Pigsty web page cross-references will use the default domain name here
endpoint: Usually used as an alternative to path, specifies the upstream server address. Setting endpoint indicates this is a reverse proxy server
In the configuration, ${admin_ip} can be used as a placeholder, and will be dynamically replaced with admin_ip during deployment
The default reverse proxy server uses endpoint.conf as the configuration template
The reverse proxy server can also configure the websocket and schema parameters
path: Usually used as an alternative to endpoint, specifies the local file server path. Setting path means this is a local web server
The local web server uses path.conf as the configuration template
The local web server can also configure the index parameter, whether to enable the file index page
certbot: Certbot certificate name, if configured, will use Certbot to apply for a certificate
If multiple servers specify the same certbot, Pigsty will merge the certificate application, and the final certificate name will be the name of this certbot
cert: Cert file path, if configured, will override the default certificate path
key: Cert key file path, if configured, will override the default certificate key path
websocket: Whether to enable WebSocket support
Only reverse proxy servers can configure this parameter, if enabled it will allow upstream to use WebSocket connections
schema: Protocol used by the upstream server, if configured, will override the default protocol
Default is http, if configured as https it will force HTTPS connections to the upstream server
index: Whether to enable file index page
Only local web servers can configure this parameter, if enabled it will enable the autoindex configuration to automatically generate index pages for directories
log: Nginx log file path
If specified, access logs will be written to this file, otherwise the default log file will be used according to the server type
Reverse proxy servers use /var/log/nginx/<name>.log as the default log file path
Configuration text directly injected into the Nginx Server configuration block
enforce_https: Redirect HTTP to HTTPS
Global configuration can be specified by nginx_sslmode: enforce
This configuration does not affect the default home server, which will always listen on ports 80 and 443 simultaneously to ensure compatibility.
REPO
This section is about the local software repository. Pigsty will create a local software repository (APT/YUM) when initializing an infra node.
During the initialization process, Pigsty will download all packages and their dependencies (specified by repo_packages) from the Internet upstream repository (specified by repo_upstream) to {{ nginx_home }} / {{ repo_name }} (default is /www/pigsty), and the total size of all dependent software is approximately 1GB.
When creating a local repository, Pigsty will skip the software download phase if the directory already exists and there is a marker file named repo_complete in the directory.
If the download speed of some packages is too slow, you can set the download proxy to complete the initial download by using the proxy_env configuration entry or directly download the pre-packaged offline package, which is essentially a local software source built on the same operating system.
repo_enabled:true# create a yum repo on this infra node?repo_home:/www # repo home dir, `/www` by defaultrepo_name:pigsty # repo name, pigsty by defaultrepo_endpoint:http://${admin_ip}:80# access point to this repo by domain or ip:portrepo_remove:true# remove existing upstream reporepo_modules:infra,node,pgsql # install upstream repo during repo bootstrap#repo_upstream: [] # where to download#repo_packages: [] # which packages to download#repo_extra_packages: [] # extra packages to downloadrepo_url_packages:[]# extra packages from url
repo_enabled
type: bool, level: G/I
create a YUM repository on this infra node? default value: true
If you have multiple infra nodes, you can disable the YUM repository on other standby nodes to reduce Internet traffic.
repo_home
type: path, level: G
repository home directory, /www by default
repo_name
type: string, level: G
repository name, pigsty by default, it is not recommended to change this value
repo_endpoint
type: url, level: G
access point to this repository by domain or IP:port, default value: http://${admin_ip}:80
If you have changed the nginx_port or nginx_ssl_port, or use a different infra node from admin node, please adjust this parameter accordingly.
The ${admin_ip} will be replaced with actual admin_ip during runtime.
If you want to keep existing upstream repository, set this value to false.
repo_modules
type: string, level: G/A
which repository modules are installed in repo_upstream, default value: infra,node,pgsql
This is a comma separated value string, it is used to filter entries in repo_upstream with corresponding module field.
For Ubuntu / Debian users, you can add redis to the list: infra,node,pgsql,redis
repo_upstream
type: upstream[], level: G
This parameter defines the upstream software repository for Pigsty. It DOES NOT have default values; you can specify it explicitly, or leave it empty if you want to use the default values.
When leaving it empty, Pigsty will use the default values from the repo_upstream_default defined in roles/node_id/vars according to your OS.
This parameter is an array of strings, with each string being a list of software packages separated by spaces, specifying which packages to be included and downloaded.
This param DOES NOT have a default value, you can specify it explicitly, or leaving it empty if you want to use the default values.
When leaving it empty, Pigsty will use the default values from the repo_packages_default defined in roles/node_id/vars according to your OS.
Each element in repo_packages will be translated into a list of package names according to the package_map defined in the above file, for specific OS distro version.
For example, on EL systems, it will be translated into:
As a convention, repo_packages usually includes software packages that are not related to the major version of PostgreSQL (such as Infra, Node, and PGDG Common), while software packages related to the major version of PostgreSQL (kernel, extensions) are usually specified in repo_extra_packages to facilitate switching between PG major versions.
repo_extra_packages
type: string[], level: G/C/I
This parameter is the same as repo_packages, but it is used for additional software packages that need to be downloaded (usually PostgreSQL version-specific packages).
The default value is an empty list. You can override it at the cluster & instance level to specify additional software packages that need to be downloaded.
If this parameter is not explicitly defined, Pigsty will load the default value from the repo_extra_packages_default defined in roles/node_id/vars, which is:
[pgsql-main ]
Each element in repo_packages will be translated into a list of package names according to the package_map defined in the above file, for specific OS distro version.
For example, on EL systems, it will be translated into:
Here $v will be replaced with the actual PostgreSQL major version number pg_version, So you can add any PG version related packages here, and Pigsty will download them for you.
repo_url_packages
type: object[] | string[], level: G
extra packages from URL, default values: []
You can use object list or string list in this parameter, in the latter case, Pigsty will use the url basename as the filename.
Note that if the region is set to china, pigsty.io will be replaced with pigsty.cc automatically.
INFRA_PACKAGE
These packages are installed on infra nodes only, including common rpm/deb/pip packages.
infra_packages
type: string[], level: G
This parameter is an array of strings, with each string being a list of common software packages separated by spaces, specifying which packages to be installed on INFRA nodes.
This parameter DOES NOT have a default value; you can specify it explicitly, or leave it empty if you want to use the default values.
When leaving it empty, Pigsty will use the default values from the repo_packages_default defined in roles/node_id/vars according to your OS.
For EL (7/8/9) system, the default values are:
infra_packages:# packages to be installed on infra nodes- grafana,loki,logcli,promtail,prometheus,alertmanager,pushgateway,grafana-plugins,restic,certbot,python3-certbot-nginx- node_exporter,blackbox_exporter,nginx_exporter,pg_exporter,pev2,nginx,dnsmasq,ansible,etcd,python3-requests,redis,mcli
For Debian (11,12) or Ubuntu (22.04, 22.04) systems, the default values are:
infra_packages:# packages to be installed on infra nodes- grafana,grafana-plugins,loki,logcli,promtail,prometheus,alertmanager,pushgateway,restic,certbot,python3-certbot-nginx- node-exporter,blackbox-exporter,nginx-exporter,pg-exporter,pev2,nginx,dnsmasq,ansible,etcd,python3-requests,redis,mcli
infra_packages_pip
type: string, level: G
pip installed packages for infra nodes, default value is empty string
NGINX
Pigsty exposes all web services through Nginx: Home Page, Grafana, Prometheus, AlertManager, etc., and other optional tools such as PGWeb, Jupyter Lab, pgAdmin, Bytebase, and other static resources & reports such as pev, schemaspy & pgbadger.
This Nginx also serves as a local YUM/APT repository.
nginx_enabled:true# enable nginx on this infra node?nginx_exporter_enabled:true# enable nginx_exporter on this infra node?nginx_sslmode:enable # nginx ssl mode? disable,enable,enforcenginx_cert_validity:397d # nginx self-signed cert validity, 397d by defaultnginx_home:/www # nginx content dir, `/www` by defaultnginx_port:80# nginx listen port, 80 by defaultnginx_ssl_port:443# nginx ssl listen port, 443 by defaultnginx_navbar:# nginx index page navigation links- {name:CA Cert ,url:'/ca.crt' ,desc:'pigsty self-signed ca.crt'}- {name:Package ,url:'/pigsty' ,desc:'local yum repo packages'}- {name:PG Logs ,url:'/logs' ,desc:'postgres raw csv logs'}- {name:Reports ,url:'/report' ,desc:'pgbadger summary report'}- {name:Explain ,url:'/pigsty/pev.html' ,desc:'postgres explain visualizer'}certbot_sign:false# sign nginx cert with certbot during setup?certbot_email:[email protected]# certbot email address, used for free sslcertbot_optionss:''# certbot extra options
nginx_enabled
type: bool, level: G/I
enable nginx on this infra node? default value: true
nginx_exporter_enabled
type: bool, level: G/I
enable nginx_exporter on this infra node? default value: true.
Setting this to false will also disable the /nginx health check stub: If your Nginx does not support the /nginx stub, you can set this value to false to disable it.
nginx_sslmode
type: enum, level: G
nginx ssl mode? which could be: disable, enable, enforce, the default value: enable
disable: listen on nginx_port and serve plain HTTP only
enforce: all links will be rendered as https:// by default
also redirect the 80 port to 443 port for all non-default servers in nginx infra_portal
nginx_cert_validity
type: duration, level: G
nginx self-signed cert validity, 397d by default
Use longer validity is not recommended, as the latest browsers require the website certificate to be valid for at most 397 days, so this is the default value.
nginx_home
type: path, level: G
nginx web server static content dir, /www by default
Nginx root directory which contains static resource and repo resource. It’s wise to set this value same as repo_home so that local repo content is automatically served.
nginx_port
type: port, level: G
Nginx listening port which serves the HTTP requests, 80 by default.
If your default 80 port is occupied or unavailable, you can consider using another port, and change repo_endpoint and repo_upstream (the local entry) accordingly.
Each record is rendered as a navigation link to the Pigsty home page App drop-down menu, and the apps are all optional, mounted by default on the Pigsty default server under http://h.pigsty/.
The url parameter specifies the URL PATH for the app, with the exception that if the ${grafana} string is present in the URL, it will be automatically replaced with the Grafana domain name defined in infra_portal.
certbot_sign
type: bool, level: G/A
sign nginx cert with certbot during setup? default value: false
When set to true, Pigsty will use certbot to automatically apply for free SSL certificates from Let’s Encrypt during the execution of infra.yml and install.yml playbooks (the nginx role).
In the infra_portal defined domain, if the certbot parameter is defined, Pigsty will use certbot to apply for the domain domain certificate, and the certificate name will be the value of the certbot parameter. If multiple servers/domains specify the same certbot parameter, Pigsty will merge and apply for a certificate for these domains, and use the value of the certbot parameter as the certificate name.
Enabling this option requires:
The current node can be accessed through the public domain name, and the DNS resolution has been correctly pointed to the public IP of the current node
The current node can access the Let’s Encrypt API interface
This option is disabled by default, you can manually execute the make cert command after installation to manually execute it, it actually calls the rendered /etc/nginx/sign-cert script, using certbot to update or apply certificates.
certbot_email
type: string, level: G/A
The email address used to receive certificate expiration reminder emails, default value: [email protected]
When certbot_sign is set to true, it is recommended to provide this parameter. Let’s Encrypt will send reminder emails to this email when the certificate is about to expire.
certbot_options
type: string, level: G/A
Additional configuration parameters passed to certbot, default value: empty string.
You can pass additional command line options to certbot through this parameter, for example --dry-run, then certbot will not actually apply for a certificate, but will preview and test it.
DNS
Pigsty will launch a default DNSMASQ server on infra nodes to serve DNS inquiry. such as h.pigstya.pigstyp.pigstyg.pigsty and sss.pigsty for optional MinIO service.
All records will be added to infra node’s /etc/hosts.d/*.
You have to add nameserver {{ admin_ip }} to your /etc/resolv to use this dns server, and node_dns_servers will do the trick.
dns_enabled:true# setup dnsmasq on this infra node?dns_port:53# dns server listen port, 53 by defaultdns_records:# dynamic dns records resolved by dnsmasq- "${admin_ip} h.pigsty a.pigsty p.pigsty g.pigsty"- "${admin_ip} api.pigsty adm.pigsty cli.pigsty ddl.pigsty lab.pigsty git.pigsty sss.pigsty wiki.pigsty"
dns_enabled
type: bool, level: G/I
setup dnsmasq on this infra node? default value: true
If you don’t want to use the default DNS server, you can set this value to false to disable it. And use node_default_etc_hosts and node_etc_hosts instead.
dns_port
type: port, level: G
dns server listen port, 53 by default
dns_records
type: string[], level: G
dynamic dns records resolved by dnsmasq, Some auxiliary domain names will be written to /etc/hosts.d/default on infra nodes by default
dns_records:# dynamic dns records resolved by dnsmasq- "${admin_ip} h.pigsty a.pigsty p.pigsty g.pigsty"- "${admin_ip} api.pigsty adm.pigsty cli.pigsty ddl.pigsty lab.pigsty git.pigsty sss.pigsty wiki.pigsty"
PROMETHEUS
Prometheus is used as the time-series database for metrics scrape, storage & analysis.
prometheus_enabled:true# enable prometheus on this infra node?prometheus_port:9058# prometheus listen port, 9058 by defaultprometheus_clean:true# clean prometheus data during init?prometheus_data:/data/prometheus# prometheus data dir, `/data/prometheus` by defaultprometheus_sd_dir:/etc/prometheus/targets# prometheus file service discovery directoryprometheus_sd_interval:5s # prometheus target refresh interval, 5s by defaultprometheus_scrape_interval:10s # prometheus scrape & eval interval, 10s by defaultprometheus_scrape_timeout:8s # prometheus global scrape timeout, 8s by defaultprometheus_options:'--storage.tsdb.retention.time=15d'# prometheus extra server optionspushgateway_enabled:true# setup pushgateway on this infra node?pushgateway_options:'--persistence.interval=1m'# pushgateway extra server optionsblackbox_enabled:true# setup blackbox_exporter on this infra node?blackbox_options:''# blackbox_exporter extra server optionsalertmanager_enabled:true# setup alertmanager on this infra node?alertmanager_port:9059# alertmanager listen port, 9059 by defaultalertmanager_options:''# alertmanager extra server optionsexporter_metrics_path:/metrics # exporter metric path, `/metrics` by defaultexporter_install:none # how to install exporter? none,yum,binaryexporter_repo_url:''# exporter repo file url if install exporter via yum
prometheus_enabled
type: bool, level: G/I
enable prometheus on this infra node?
default value: true
prometheus_port
type: port, level: G
The listening port for Prometheus, default value is 9058.
Newly added since 3.7, with previous default value changing from 9090 to 9058.
Because the EL10 distro has a new default web service cockpit-ws which also uses port 9090.
prometheus_clean
type: bool, level: G/A
clean prometheus data during init? default value: true
Extra cli args for prometheus server, the default value will set up a 15-day data retention to limit disk usage.
pushgateway_enabled
type: bool, level: G/I
setup pushgateway on this infra node? default value: true
pushgateway_options
type: arg, level: G
pushgateway extra server options, default value: --persistence.interval=1m
blackbox_enabled
type: bool, level: G/I
setup blackbox_exporter on this infra node? default value: true
blackbox_options
type: arg, level: G
blackbox_exporter extra server options, default value is empty string
alertmanager_enabled
type: bool, level: G/I
setup alertmanager on this infra node? default value: true
alertmanager_port
type: port, level: G
The listening port for AlertManager, default value is 9059.
The default value change from 9093 since v3.7.
Because the kafka’s default peer port is also 9093
alertmanager_options
type: arg, level: G
alertmanager extra server options, default value is empty string
exporter_metrics_path
type: path, level: G
exporter metric path, /metrics by default
exporter_install
type: enum, level: G
(OBSOLETE) how to install exporter? none,yum,binary
default value: none
Specify how to install Exporter:
none: No installation, (by default, the Exporter has been previously installed by the node_pkg task)
yum: Install using yum (if yum installation is enabled, run yum to install node_exporter and pg_exporter before deploying Exporter)
binary: Install using a copy binary (copy node_exporter and pg_exporter binary directly from local file dir, not recommended)
When installing with yum, if exporter_repo_url is specified (not empty), the installation will first install the REPO file under that URL into /etc/yum.repos.d. This feature allows you to install Exporter directly without initializing the node infrastructure. It is not recommended for regular users to use binary installation. This mode is usually used for emergency troubleshooting and temporary problem fixes.
(OBSOLETE) exporter repo file url if install exporter via yum
default value is empty string
Default is empty; when exporter_install is yum, the repo specified by this parameter will be added to the node source list.
GRAFANA
Grafana is the visualization platform for Pigsty’s monitoring system.
It can also be used as a low code data visualization environment
grafana_enabled:true# enable grafana on this infra node?grafana_clean:true# clean grafana data during init?grafana_admin_username:admin # grafana admin username, `admin` by defaultgrafana_admin_password:pigsty # grafana admin password, `pigsty` by defaultloki_enabled:true# enable loki on this infra node?loki_clean:false# whether remove existing loki data?loki_data:/data/loki # loki data dir, `/data/loki` by defaultloki_retention:15d # loki log retention period, 15d by default
grafana_enabled
type: bool, level: G/I
enable grafana on this infra node? default value: true
grafana_clean
type: bool, level: G/A
clean grafana data during init? default value: true
grafana_admin_username
type: username, level: G
grafana admin username, admin by default
grafana_admin_password
type: password, level: G
grafana admin password, pigsty by default
default value: pigsty
WARNING: Change this to a strong password before deploying to production environment
LOKI
loki_enabled
type: bool, level: G/I
enable loki on this infra node? default value: true
Manage infra components, local repo, nginx portal, and domain names
Here are some admin tasks related to the INFRA module
Nginx Portal
Nginx Portal for WebUI services
Local Repo
Manage local APT / YUM repository
Domain Name
Use local / public domain names
CA & Cert
Use self-signed or real HTTPS certificates
Install INFRA
Use the infra.yml playbook to install the INFRA module on Infra nodes:
./infra.yml # Install INFRA module on infra group
Remove INFRA
Use the infra-rm.yml playbook to uninstall the INFRA module from Infra nodes:
./infra-rm.yml # Uninstall INFRA module from infra group
Expand INFRA
To scale out an existing INFRA deployment, first modify the infra group by adding new node IPs and assigning unique infra_seq numbers:
all:children:infra:hosts:10.10.10.10:{infra_seq:1}# Existing node #110.10.10.11:{infra_seq:2}# New node #2 (fresh meat!)
Then use the infra.yml playbook to install INFRA on the new node:
./infra.yml -l 10.10.10.11 # Install INFRA on the new node
Local Repo
Use these playbook tasks to manage local package repositories (YUM/APT) on Infra nodes :
./infra.yml -t repo # Create local repository from internet or offline packages./infra.yml -t repo_dir # Create local repository directory./infra.yml -t repo_check # Check if local repository exists./infra.yml -t repo_prepare # Use existing local repository if available./infra.yml -t repo_build # Build local repository from upstream if not exists./infra.yml -t repo_upstream # Add upstream repository/list files./infra.yml -t repo_remove # Remove existing repository files if repo_remove=true./infra.yml -t repo_add # Add upstream repository files to /etc/yum.repos.d (or apt)./infra.yml -t repo_url_pkg # Download packages defined in repo_url_packages./infra.yml -t repo_cache # Create metadata cache with yum makecache / apt update./infra.yml -t repo_boot_pkg # Install bootstrap packages (createrepo_c, yum-utils, etc)./infra.yml -t repo_pkg # Download packages & deps from upstream./infra.yml -t repo_create # Create local repository with createrepo_c / dpkg-dev./infra.yml -t repo_use # Add new repository to /etc/yum.repos.d | apt sources./infra.yml -t repo_nginx # Start nginx as file server if not running
Commonly used commands:
./infra.yml -t repo_upstream # Add upstream repositories defined in repo_upstream./infra.yml -t repo_pkg # Download packages and their dependencies./infra.yml -t repo_create # Create/update local YUM/APT repository
There are three playbooks related to the INFRA module:
infra.yml: Initialize Pigsty infrastructure on infra nodes
infra-rm.yml: Remove infrastructure components from infra nodes
install.yml: Perform a complete one-time installation of Pigsty on the current node
infra.yml
The INFRA module playbook infra.yml initializes the infrastructure module on Infra nodes defined in the infra group of your config file.
This playbook performs the following tasks:
Configures directories and environment variables on Infra nodes
Downloads and creates a local software repository to accelerate subsequent installations (skipped if offline packages are used or if a local repository already exists)
Incorporates the current Infra node as a common node managed by Pigsty
Deploys infrastructure components, including Prometheus, Grafana, Loki, Alertmanager, PushGateway, Blackbox Exporter, etc.
This playbook executes on the infra group by default:
Pigsty installs the INFRA module on the group named infra in the config file
During configure, Pigsty marks the current installation node as an Infra node and replaces the placeholder IP address 10.10.10.10 in the config template with the current node’s primary IP address
This node can initiate management tasks and host infrastructure components while functioning like any regular managed node
Playbook notes:
This is an idempotent playbook - repeated execution will overwrite infra components on Infra nodes
Unless prometheus_clean is set to false, Prometheus time series metrics will be lost
Unless loki_clean is set to false, Loki log data will be lost
Unless grafana_clean is set to false, Grafana dashboards and configuration changes will be lost
When the local software repo /www/pigsty/repo_complete exists, this playbook skips downloading software from the internet
Complete execution takes approximately 1 ~ 3 minutes, depending on machine configuration and network conditions
Downloading software directly from the original upstream sources (without using offline packages) may take 5-10 minutes, depending on your network connection
Demo
Available Tasks
Here’s the list of available tasks in the infra.yml playbook:
#--------------------------------------------------------------## Tasks#--------------------------------------------------------------## ca : create self-signed CA in localhost files/pki# - ca_dir : create CA directory# - ca_private : generate CA private key: files/pki/ca/ca.key# - ca_cert : sign CA certificate: files/pki/ca/ca.crt## id : generate node identity## repo : bootstrap a local YUM repository from internet or offline packages# - repo_dir : create repository directory# - repo_check : check repository exists# - repo_prepare : use existing repository if exists# - repo_build : build repository from upstream if not exists# - repo_upstream : handle upstream repository files in /etc/yum.repos.d# - repo_remove : remove existing repository file if repo_remove == true# - repo_add : add upstream repository files to /etc/yum.repos.d# - repo_url_pkg : download packages from internet defined by repo_url_packages# - repo_cache : make upstream YUM cache with yum makecache# - repo_boot_pkg : install bootstrap packages such as createrepo_c, yum-utils, etc.# - repo_pkg : download packages & dependencies from upstream repository# - repo_create : create a local YUM repository with createrepo_c & modifyrepo_c# - repo_use : add newly built repository into /etc/yum.repos.d# - repo_nginx : launch nginx for repository if no nginx is serving## node/haproxy/docker/monitor : set up infra node as a common node (check node.yml)# - node_name, node_hosts, node_resolv, node_firewall, node_ca, node_repo, node_pkg# - node_feature, node_kernel, node_tune, node_sysctl, node_profile, node_ulimit# - node_data, node_admin, node_timezone, node_ntp, node_crontab, node_vip# - haproxy_install, haproxy_config, haproxy_launch, haproxy_reload# - docker_install, docker_admin, docker_config, docker_launch, docker_image# - haproxy_register, node_exporter, node_register, promtail## infra : set up infra components# - infra_env : env_dir, env_pg, env_pgadmin, env_var# - infra_pkg : infra_pkg_yum, infra_pkg_pip# - infra_user : set up infra OS user group# - infra_cert : issue certificate for infra components# - dns : dns_config, dns_record, dns_launch# - nginx : nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_certbot, nginx_reload, nginx_exporter# - prometheus : prometheus_clean, prometheus_dir, prometheus_config, prometheus_launch, prometheus_reload# - alertmanager : alertmanager_config, alertmanager_launch# - pushgateway : pushgateway_config, pushgateway_launch# - blackbox : blackbox_config, blackbox_launch# - grafana : grafana_clean, grafana_config, grafana_launch, grafana_provision# - loki : loki clean, loki_dir, loki_config, loki_launch# - infra_register : register infra components to prometheus#--------------------------------------------------------------#
infra-rm.yml
The INFRA module playbook infra-rm.yml removes Pigsty infrastructure from Infra nodes defined in the infra group of your config file.
Common subtasks include:
./infra-rm.yml # Remove the INFRA module./infra-rm.yml -t service # Stop infrastructure services on INFRA./infra-rm.yml -t data # Remove retained data on INFRA./infra-rm.yml -t package # Uninstall packages installed on INFRA
install.yml
The INFRA module playbook install.yml performs a complete one-time installation of Pigsty on all nodes.
Ansible for automation, deployment, and administration;
Nginx for exposing any WebUI service and serving the YUM/APT repository;
Self-Signed CA for SSL/TLS certificates;
Prometheus for monitoring metrics
Grafana for monitoring/visualization
Loki for logging collection
AlertManager for alerts aggregation
Chronyd for NTP time synchronization on the admin node.
DNSMasq for DNS registration and resolution.
ETCD as DCS for PostgreSQL HA (dedicated module);
PostgreSQL on meta nodes as CMDB (optional);
Docker for stateless applications & tools (optional).
How to restore Prometheus targets
If you accidentally deleted the Prometheus targets dir, you can register monitoring targets to Prometheus again with the:
./infra.yml -t register_prometheus # register all infra targets to prometheus on infra nodes./node.yml -t register_prometheus # register all node targets to prometheus on infra nodes./etcd.yml -t register_prometheus # register all etcd targets to prometheus on infra nodes./minio.yml -t register_prometheus # register all minio targets to prometheus on infra nodes./pgsql.yml -t register_prometheus # register all pgsql targets to prometheus on infra nodes
How to restore Grafana datasource
PGSQL Databases in pg_databases are registered as Grafana datasource by default.
If you accidentally deleted the registered postgres datasource in Grafana, you can register them again with
./pgsql.yml -t register_grafana # register all pgsql database (in pg_databases) as grafana datasource
How to restore the HAProxy admin page proxy
The haproxy admin page is proxied by Nginx under the default server.
If you accidentally deleted the registered haproxy proxy settings in /etc/nginx/conf.d/haproxy, you can restore them again with
./node.yml -t register_nginx # register all haproxy admin page proxy settings to nginx on infra nodes
How to restore the DNS registration
PGSQL cluster/instance domain names are registered to /etc/hosts.d/<name> on infra nodes by default.
You can restore them with the following command:
./pgsql.yml -t pg_dns # register pg DNS names to dnsmasq on infra nodes
How to expose a new Nginx upstream service
If you wish to expose a new WebUI service via the Nginx portal, you can add the service definition to the infra_portal parameter.
And re-run ./infra.yml -t nginx_config,nginx_launch to update & apply the Nginx configuration.
If you wish to access with HTTPS, you must remove files/pki/csr/pigsty.csr, files/pki/nginx/pigsty.{key,crt} to force re-generating the Nginx SSL/TLS certificate to include the new upstream’s domain name.
How to expose a web service through Nginx?
While you can directly access services via IP:Port, we still recommend consolidating access points by using domain names and uniformly accessing various web-based services through the Nginx portal. This approach helps centralize access, reduce the number of exposed ports, and facilitates access control and auditing.
If you wish to expose a new WebUI service through the Nginx portal, you can add the service definition to the infra_portal parameter. For example, here is the config used by the public demo site, which exposes several additional web services:
After completing the Nginx upstream service definition, use the following commands to register the new service with Nginx.
./infra.yml -t nginx_config # regenerate Nginx config./infra.yml -t nginx_launch # update and apply nginx config# you can reload nginx with ansibleansible infra -b -a 'nginx -s reload'# reload nginx with ansible
If you wish to access via HTTPS, you must delete files/pki/csr/pigsty.csr and files/pki/nginx/pigsty.{key,crt} to force the regeneration of the Nginx SSL/TLS certificate to include the new upstream domain names. If you prefer to use an SSL certificate issued by an authoritative organization instead of a certificate issued by Pigsty’s self-signed CA, you can place it in the /etc/nginx/conf.d/cert/ directory and modify the corresponding configuration: /etc/nginx/conf.d/<name>.conf.
How to manually add upstream repo files
Pigsty has a built-in wrapper script bin/repo-add, which will invoke Ansible playbook node.yml to add repository files to corresponding nodes.
bin/repo-add <selector> [modules]bin/repo-add 10.10.10.10 # add node repos for node 10.10.10.10bin/repo-add infra node,infra # add node and infra repos for group infrabin/repo-add infra node,local # add node repos and local pigsty repobin/repo-add pg-test node,pgsql # add node & pgsql repos for group pg-test
13 - Node
Enroll linux machine hosts into desired state
To deploy modules, you’ll have to enroll your linux servers into pigsty by installing the NODE module on them.
Architecture
Architecture, core concepts, identity management
Configuration
Configure node identities
Parameters
Customize node components with 64 parameters
Administration
Setup VIP, manage monitoring node targets
Playbooks
Ansible playbooks that can be used in node module
Monitoring
Dashboards, metrics, record & alerting rules.
FAQ
Frequently Asked Questions about node module
13.1 - Architecture
Node types, architecture, and core concepts
A "node" refers to a resource that is SSH accessible and offers a bare Linux OS environment.
It could be a physical machine, a virtual machine, or an OS-like container equipped with systemd, sudo and sshd.
There are three different types of nodes in Pigsty, In a one-node deployment, they are the same one.
Any nodes managed by Pigsty, including admin & infra nodes
Example
In the following 4-node sandbox config snippet, we have 4 common nodes.
And the 10.10.10.10 is marked as Infra Node and Admin Node simultaneously.
all:children:infra:{hosts:{10.10.10.10:{infra_seq:1}}}# <--- mark this as infra nodeetcd:{hosts:{10.10.10.10:{etcd_seq:1 } }, vars:{etcd_cluster:etcd } }pg-meta:{hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }, vars:{pg_cluster:pg-meta } }pg-test:hosts:10.10.10.11:{pg_seq:1, pg_role:primary }10.10.10.12:{pg_seq:2, pg_role:replica }10.10.10.13:{pg_seq:3, pg_role:replica }vars:{pg_cluster:pg-test }vars:admin_ip:10.10.10.10# <--- mark this as admin node
Common Node
Common nodes have the following components enabled by default in Pigsty:
Component
Port
Description
Default
node_exporter
9100
Node Monitoring Metrics Exporter
Enabled
haproxy
9101
HAProxy admin / metrics port
Enabled
promtail
9080
Log collecting agent
Enabled
These components are optional, can be enabled with parameters.
Component
Port
Description
Default
docker
9323
Enable Container Service
Disabled
keepalived
N/A
Manage Node Cluster L2 VIP
Disabled
keepalived_exporter
9650
Monitoring Keepalived Status
Disabled
ADMIN Node
Admin Node is the first node where Pigsty is installed, all control commands are issued from it.
There is one and only one admin node in a pigsty deployment, which is specified by admin_ip.
This parameter will be set during the configure procedure, to the value of primary IP address.
Secure admin node access
The admin node should have nopass ssh and sudo access to all other nodes in the environment,
which is dangerous when exposed to unauthorized access. Please keep it safe.
Admin node is the first node where Pigsty is installed
The admin node is the first node where Pigsty is installed, and it is used to issue commands to other nodes.
The admin node is usually overlapped with the infra node, same as the first infra node
It is possible to use your local laptop as admin node
It is possible to install pigsty on your local laptop / macbook, install ansible and issue commands from there.
INFRA Node
A pigsty deployment may have one or more infra nodes, at least 1, 2 or more for production deployment.
The infra group specifies infra nodes in the inventory.
And infra nodes will have INFRA module installed (DNS, Nginx, Prometheus, Grafana, etc…).
Component
Port
Domain
Description
nginx
80
h.pigsty
Web Service Portal (YUM/APT Repo)
alertmanager
9059
a.pigsty
Alert Aggregation and delivery
prometheus
9058
p.pigsty
Monitoring Time Series Database
grafana
3000
g.pigsty
Visualization Platform
lok
3100
-
Logging Collection Server
pushgateway
9091
-
Collect One-Time Job Metrics
blackbox_exporter
9115
-
Blackbox Probing
dnsmasq
53
-
DNS Server
chronyd
123
-
NTP Time Server
ansible
-
-
Run playbooks
PGSQL Node
The node with PGSQL module installed is called a PGSQL node. The node and PG instance are 1:1 deployed.
And node identities can be borrowed from pg instances with node_id_from_pg option.
Here are additional components & service port on PGSQL nodes:
Component
Port
Description
Status
postgres
5432
PostgreSQL Server Process Managed by Patroni
Enabled
pgbouncer
6432
Pgbouncer Connection Pool
Enabled
patroni
8008
Patroni HA Component, Manage postgres
Enabled
primary @ haproxy
5433
Primary connection pool: Read/Write Service
Enabled
replica @ haproxy
5434
Replica connection pool: Read-only Service
Enabled
default @ haproxy
5436
Primary Direct Connect Service
Enabled
offline @ haproxy
5438
Offline Direct Connect: Offline Read Service
Enabled
pg_exporter
9630
postgres Monitoring Metrics Exporter
Enabled
pgbouncer_exporter
9631
pgbouncer Monitoring Metrics Exporter
Enabled
pgbackrest_exporter
9854
pgbackrest Monitoring Metrics Exporter
Enabled
vip-manager
-
Bind VIP to the primary
Disabled
13.2 - Configure
node identity, dns, vip, data dir, and haproxy services
You don’t need to define node cluster and node instances explicitly.
It is often implied by other database module’s cluster definition.
If you define a PGSQL cluster, it defines a node cluster implicitly.
But there are some cases where you may want to use explicit named node clusters / instances.
Such as running node groups for dedicated purpose (e.g., haproxy groups, node buffer pool, etc…)
Identity Parameters
Pigsty uses node’s primary IPv4 address (inventory_hostname) as its identity.
This can be useful when you just want to monitor these nodes rather than running database on them.
Borrowed Identity
Because Pigsty is using a 1:1 mapping between NODE and PG instance (there’s only one PG instance per node),
the node’s identity can be borrowed from corresponding PG instances:
This is default behavior, it’s convenient to have PG & NODE with the same cls / ins identity labels in the monitoring system.
This can be disabled by overwriting the node_id_from_pg param to false
If there’s no corresponding PG instance defined, and no explicit nodename & node_cluster defined,
the node will be labeled with cls as nodes and ins as the current hostname.
SSH Connection
The inventory_hostname is used by Ansible to connect to the node via SSH.
If your node cannot be simply access via ssh <inventory_hostname>, ssh alias and more ansible conn param (even a different IP) can be used.
But the inventory_hostname is still the core identity of the node.
The Node module tunes target nodes into the desired state and integrates them into the Pigsty monitoring system.
NODE_ID
Each node has identity parameters that are configured through the parameters in <cluster>.hosts and <cluster>.vars. Check NODE Identity for details.
nodename
name: nodename, type: string, level: I
node instance identity, use hostname if missing, optional
no default value, Null or empty string means nodename will be set to node’s current hostname.
If node_id_from_pg is true (by default) and nodename is not explicitly defined, nodename will try to use ${pg_cluster}-${pg_seq} first, if PGSQL is not defined on this node, it will fall back to default HOSTNAME.
If nodename_overwrite is true, the node name will also be used as the HOSTNAME.
node_cluster
name: node_cluster, type: string, level: C
node cluster identity, use ’nodes’ if missing, optional
default values: nodes
If node_id_from_pg is true (by default) and node_cluster is not explicitly defined, node_cluster will try to use ${pg_cluster} first, if PGSQL is not defined on this node, it will fall back to default HOSTNAME.
nodename_overwrite
name: nodename_overwrite, type: bool, level: C
overwrite node’s hostname with nodename?
default value is true, a non-empty node name nodename will override the hostname of the current node.
When the nodename parameter is undefined or an empty string, but node_id_from_pg is true, the node name will try to use {{ pg_cluster }}-{{ pg_seq }}, borrow identity from the 1:1 PostgreSQL Instance’s ins name.
No changes are made to the hostname if the nodename is undefined, empty, or an empty string and node_id_from_pg is false.
nodename_exchange
name: nodename_exchange, type: bool, level: C
exchange nodename among play hosts?
default value is false
When this parameter is enabled, node names are exchanged between the same group of nodes executing the node.yml playbook, written to /etc/hosts.
node_id_from_pg
name: node_id_from_pg, type: bool, level: C
use postgres identity as node identity if applicable?
default value is true
Borrow PostgreSQL cluster & instance identity if applicable.
It’s useful to use same identity for postgres & node if there’s a 1:1 relationship
NODE_DNS
Pigsty configs static DNS records and dynamic DNS resolver for nodes.
If you already have a DNS server, set node_dns_method to none to disable dynamic DNS setup.
node_write_etc_hosts:true# modify `/etc/hosts` on target node?node_default_etc_hosts:# static dns records in `/etc/hosts`- "${admin_ip} h.pigsty a.pigsty p.pigsty g.pigsty"node_etc_hosts:[]# extra static dns records in `/etc/hosts`node_dns_method:add # how to handle dns servers: add,none,overwritenode_dns_servers:['${admin_ip}']# dynamic nameserver in `/etc/resolv.conf`node_dns_options:# dns resolv options in `/etc/resolv.conf`- options single-request-reopen timeout:1
node_default_etc_hosts is an array. Each element is a DNS record with format <ip> <name>.
It is used for global static DNS records. You can use node_etc_hosts for ad hoc records for each cluster.
Make sure to write a DNS record like 10.10.10.10 h.pigsty a.pigsty p.pigsty g.pigsty to /etc/hosts to ensure that the local yum repo can be accessed using the domain name before the DNS Nameserver starts.
add: Append the records in node_dns_servers to /etc/resolv.conf and keep the existing DNS servers. (default)
overwrite: Overwrite /etc/resolv.conf with the record in node_dns_servers
none: If a DNS server is provided in the production env, the DNS server config can be skipped.
node_dns_servers
name: node_dns_servers, type: string[], level: C
dynamic nameserver in /etc/resolv.conf
default values: ["${admin_ip}"] , the default nameserver on admin node will be added to /etc/resolv.conf as the first nameserver.
node_dns_options
name: node_dns_options, type: string[], level: C
dns resolv options in /etc/resolv.conf, default value:
- options single-request-reopen timeout:1
NODE_PACKAGE
This section is about upstream yum repos & packages to be installed.
node_repo_modules:local # upstream repo to be added on node, local by defaultnode_repo_remove:true# remove existing repo on node?node_packages:[openssh-server] # packages to be installed current nodes with the latest version#node_default_packages: [] # default packages to be installed on infra nodes, (defaults are load from node_id/vars)
node_repo_modules
name: node_repo_modules, type: string, level: C/A
upstream repo to be added on node, default value: local
This parameter specifies the upstream repo to be added to the node. It is used to filter the repo_upstream entries and only the entries with the same module value will be added to the node’s software source. Which is similar to the repo_modules parameter.
node_repo_remove
name: node_repo_remove, type: bool, level: C/A
remove existing repo on node?
default value is true, and thus Pigsty will move existing repo file in /etc/yum.repos.d to a backup dir: /etc/yum.repos.d/backup before adding upstream repos On Debian/Ubuntu, Pigsty will backup & move /etc/apt/sources.list(.d) to /etc/apt/backup.
node_packages
name: node_packages, type: string[], level: C
packages to be installed current nodes, default values: [openssh-server].
Each element is a comma-separated list of package names, which will be installed on the current node in addition to node_default_packages
Packages specified in this parameter will be upgraded to the latest version, and the default value is [openssh-server], which will upgrade sshd by default to avoid SSH CVE.
This parameter is usually used to install additional software packages that are ad hoc for the current node/cluster.
node_default_packages
name: node_default_packages, type: string[], level: G
default packages to be installed on all nodes, the default values is not defined.
This param is an array of strings, each string is a comma-separated list of package names, which will be installed on all nodes by default.
This param DOES NOT have a default value, you can specify it explicitly, or leave it empty if you want to use the default values.
When leaving it empty, Pigsty will use the default values from the node_packages_default defined in roles/node_id/vars according to you OS.
Configure tuned templates, features, kernel modules, sysctl params on node.
node_disable_firewall:true# disable node firewall? true by defaultnode_disable_selinux:true# disable node selinux? true by defaultnode_disable_numa:false# disable node numa, reboot requirednode_disable_swap:false# disable node swap, use with cautionnode_static_network:true# preserve dns resolver settings after rebootnode_disk_prefetch:false# setup disk prefetch on HDD to increase performancenode_kernel_modules:[softdog, ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh ]node_hugepage_count:0# number of 2MB hugepage, take precedence over rationode_hugepage_ratio:0# node mem hugepage ratio, 0 disable it by defaultnode_overcommit_ratio:0# node mem overcommit ratio, 0 disable it by defaultnode_tune:oltp # node tuned profile: none,oltp,olap,crit,tinynode_sysctl_params:{}# sysctl parameters in k:v format in addition to tuned
node_disable_firewall
name: node_disable_firewall, type: bool, level: C
disable node firewall? true by default
default value is true
node_disable_selinux
name: node_disable_selinux, type: bool, level: C
disable node selinux? true by default
default value is true
node_disable_numa
name: node_disable_numa, type: bool, level: C
disable node numa, reboot required
default value is false
Boolean flag, default is not off. Note that turning off NUMA requires a reboot of the machine before it can take effect!
If you don’t know how to set the CPU affinity, it is recommended to turn off NUMA.
node_disable_swap
name: node_disable_swap, type: bool, level: C
disable node swap, use with caution
default value is false
Turning off SWAP is not recommended. However, SWAP should be disabled when your node is used for a Kubernetes deployment.
If there is enough memory and the database is deployed exclusively, it may slightly improve performance
node_static_network
name: node_static_network, type: bool, level: C
preserve dns resolver settings after reboot, default value is true
Enabling static networking means that machine reboots will not overwrite your DNS Resolv config with NIC changes. It is recommended to enable it in production environment.
node_disk_prefetch
name: node_disk_prefetch, type: bool, level: C
setup disk prefetch on HDD to increase performance
default value is false, Consider enable this when using HDD.
node_kernel_modules
name: node_kernel_modules, type: string[], level: C
For example, if you have default 25% mem for postgres shard buffers, you can set this value to 0.27 ~ 0.30, Wasted hugepage can be reclaimed later with /pg/bin/pg-tune-hugepage
node_overcommit_ratio
name: node_overcommit_ratio, type: int, level: C
node mem overcommit ratio, 0 disable it by default. this is an integer from 0 to 100+ .
default values: 0, which will set vm.overcommit_memory=0, otherwise vm.overcommit_memory=2 will be used, and this value will be used as vm.overcommit_ratio.
It is recommended to set use a vm.overcommit_ratio on dedicated pgsql nodes. e.g. 50 ~ 100.
oltp: Regular OLTP templates with optimized latency
olap : Regular OLAP templates to optimize throughput
crit: Core financial business templates, optimizing the number of dirty pages
Usually, the database tuning template pg_conf should be paired with the node tuning template: node_tune
node_sysctl_params
name: node_sysctl_params, type: dict, level: C
sysctl parameters in k:v format in addition to tuned
default values: {}
Dictionary K-V structure, Key is kernel sysctl parameter name, Value is the parameter value.
You can also define sysctl parameters with tuned profile
NODE_ADMIN
This section is about admin users and it’s credentials.
node_data:/data # node main data directory, `/data` by defaultnode_admin_enabled:true# create a admin user on target node?node_admin_uid:88# uid and gid for node admin usernode_admin_username:dba # name of node admin user, `dba` by defaultnode_admin_ssh_exchange:true# exchange admin ssh key among node clusternode_admin_pk_current:true# add current user's ssh pk to admin authorized_keysnode_admin_pk_list:[]# ssh public keys to be added to admin user
node_data
name: node_data, type: path, level: C
node main data directory, /data by default
default values: /data
If specified, this path will be used as major data disk mountpoint. And a dir will be created and throwing a warning if path not exists.
The data dir is owned by root with mode 0777.
node_admin_enabled
name: node_admin_enabled, type: bool, level: C
create a admin user on target node?
default value is true
Create an admin user on each node (password-free sudo and ssh), an admin user named dba (uid=88) will be created by default, which can access other nodes in the env and perform sudo from the meta node via SSH password-free.
node_admin_uid
name: node_admin_uid, type: int, level: C
uid and gid for node admin user
default values: 88
node_admin_username
name: node_admin_username, type: username, level: C
name of node admin user, dba by default
default values: dba
node_admin_ssh_exchange
name: node_admin_ssh_exchange, type: bool, level: C
exchange admin ssh key among node cluster
default value is true
When enabled, Pigsty will exchange SSH public keys between members during playbook execution, allowing admins node_admin_username to access each other from different nodes.
node_admin_pk_current
name: node_admin_pk_current, type: bool, level: C
add current user’s ssh pk to admin authorized_keys
default value is true
When enabled, on the current node, the SSH public key (~/.ssh/id_rsa.pub) of the current user is copied to the authorized_keys of the target node admin user.
When deploying in a production env, be sure to pay attention to this parameter, which installs the default public key of the user currently executing the command to the admin user of all machines.
node_admin_pk_list
name: node_admin_pk_list, type: string[], level: C
ssh public keys to be added to admin user
default values: []
Each element of the array is a string containing the key written to the admin user ~/.ssh/authorized_keys, and the user with the corresponding private key can log in as an admin user.
When deploying in production envs, be sure to note this parameter and add only trusted keys to this list.
node_aliases
name: node_aliases, type: dict, level: C/I
extra aliases to be added to admin user’s shell profile
default values: {}
You can add extra shell aliases to it, pigsty will add these aliases to the /etc/profile.d/node.alias.sh file on the target node:
node_aliases:g:gitd:docker
This will generate:
aliasg="git"aliasd="docker"
NODE_TIME
node_timezone:''# setup node timezone, empty string to skipnode_ntp_enabled:true# enable chronyd time sync service?node_ntp_servers:# ntp servers in `/etc/chrony.conf`- pool pool.ntp.org iburstnode_crontab_overwrite:true# overwrite or append to `/etc/crontab`?node_crontab:[]# crontab entries in `/etc/crontab`
node_timezone
name: node_timezone, type: string, level: C
setup node timezone, empty string to skip
default value is empty string, which will not change the default timezone (usually UTC)
node_ntp_enabled
name: node_ntp_enabled, type: bool, level: C
enable chronyd time sync service?
default value is true, and thus Pigsty will override the node’s /etc/chrony.conf by with node_ntp_servers.
If you already a NTP server configured, just set to false to leave it be.
node_ntp_servers
name: node_ntp_servers, type: string[], level: C
ntp servers in /etc/chrony.conf, default value: ["pool pool.ntp.org iburst"]
You can use ${admin_ip} to sync time with ntp server on admin node rather than public ntp server.
node_ntp_servers:['pool ${admin_ip} iburst']
node_crontab_overwrite
name: node_crontab_overwrite, type: bool, level: C
overwrite or append to /etc/crontab?
default value is true, and pigsty will render records in node_crontab in overwrite mode rather than appending to it.
node_crontab
name: node_crontab, type: string[], level: C
crontab entries in /etc/crontab
default values: []
NODE_VIP
You can bind an optional L2 VIP among one node cluster, which is disabled by default.
L2 VIP can only be used in same L2 LAN, which may incurs extra restrictions on your network topology.
If enabled, You have to manually assign the vip_address and vip_vrid for each node cluster.
It is user’s responsibility to ensure that the address / vrid is unique among the same LAN.
vip_enabled:false# enable vip on this node cluster?# vip_address: [IDENTITY] # node vip address in ipv4 format, required if vip is enabled# vip_vrid: [IDENTITY] # required, integer, 1-254, should be unique among same VLANvip_role:backup # optional, `master/backup`, backup by default, use as init rolevip_preempt:false# optional, `true/false`, false by default, enable vip preemptionvip_interface:eth0 # node vip network interface to listen, `eth0` by defaultvip_dns_suffix:''# node vip dns name suffix, empty string by defaultvip_exporter_port:9650# keepalived exporter listen port, 9650 by default
vip_enabled
name: vip_enabled, type: bool, level: C
enable vip on this node cluster? default value is false, means no L2 VIP is created for this node cluster.
L2 VIP can only be used in same L2 LAN, which may incurs extra restrictions on your network topology.
vip_address
name: vip_address, type: ip, level: C
node vip address in IPv4 format, required if node vip_enabled.
no default value. This parameter must be explicitly assigned and unique in your LAN.
vip_vrid
name: vip_vrid, type: int, level: C
integer, 1-254, should be unique in same VLAN, required if node vip_enabled.
no default value. This parameter must be explicitly assigned and unique in your LAN.
vip_role
name: vip_role, type: enum, level: I
node vip role, could be master or backup, will be used as initial keepalived state.
vip_preempt
name: vip_preempt, type: bool, level: C/I
optional, true/false, false by default, enable vip preemption
default value is false, means no preempt is happening when a backup have higher priority than living master.
vip_interface
name: vip_interface, type: string, level: C/I
node vip network interface to listen, eth0 by default.
It should be the same primary intranet interface of your node, which is the IP address you used in the inventory file.
If your node have different interface, you can override it on instance vars
vip_dns_suffix
name: vip_dns_suffix, type: string, level: C/I
node vip dns name suffix, empty string by default. It will be used as the DNS name of the node VIP.
vip_exporter_port
name: vip_exporter_port, type: port, level: C/I
keepalived exporter listen port, 9650 by default.
HAPROXY
HAProxy is installed on every node by default, exposing services in a NodePort manner.
haproxy_enabled:true# enable haproxy on this node?haproxy_clean:false# cleanup all existing haproxy config?haproxy_reload:true# reload haproxy after config?haproxy_auth_enabled:true# enable authentication for haproxy admin pagehaproxy_admin_username:admin # haproxy admin username, `admin` by defaulthaproxy_admin_password:pigsty # haproxy admin password, `pigsty` by defaulthaproxy_exporter_port:9101# haproxy admin/exporter port, 9101 by defaulthaproxy_client_timeout:24h # client side connection timeout, 24h by defaulthaproxy_server_timeout:24h # server side connection timeout, 24h by defaulthaproxy_services:[]# list of haproxy service to be exposed on node
haproxy_enabled
name: haproxy_enabled, type: bool, level: C
enable haproxy on this node?
default value is true
haproxy_clean
name: haproxy_clean, type: bool, level: G/C/A
cleanup all existing haproxy config?
default value is false
haproxy_reload
name: haproxy_reload, type: bool, level: A
reload haproxy after config?
default value is true, it will reload haproxy after config change.
If you wish to check before apply, you can turn off this with cli args and check it.
haproxy_auth_enabled
name: haproxy_auth_enabled, type: bool, level: G
enable authentication for haproxy admin page
default value is true, which will require a http basic auth for admin page.
disable it is not recommended, since your traffic control will be exposed
haproxy_admin_username
name: haproxy_admin_username, type: username, level: G
haproxy admin username, admin by default
haproxy_admin_password
name: haproxy_admin_password, type: password, level: G
haproxy admin password, pigsty by default
PLEASE CHANGE IT IN YOUR PRODUCTION ENVIRONMENT!
haproxy_exporter_port
name: haproxy_exporter_port, type: port, level: C
haproxy admin/exporter port, 9101 by default
haproxy_client_timeout
name: haproxy_client_timeout, type: interval, level: C
client side connection timeout, 24h by default
haproxy_server_timeout
name: haproxy_server_timeout, type: interval, level: C
server side connection timeout, 24h by default
haproxy_services
name: haproxy_services, type: service[], level: C
list of haproxy service to be exposed on node, default values: []
Each element is a service definition, here is an ad hoc haproxy service example:
haproxy_services:# list of haproxy service# expose pg-test read only replicas- name:pg-test-ro # [REQUIRED] service name, uniqueport:5440# [REQUIRED] service port, uniqueip:"*"# [OPTIONAL] service listen addr, "*" by defaultprotocol:tcp # [OPTIONAL] service protocol, 'tcp' by defaultbalance:leastconn # [OPTIONAL] load balance algorithm, roundrobin by default (or leastconn)maxconn:20000# [OPTIONAL] max allowed front-end connection, 20000 by defaultdefault:'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'options:- option httpchk- option http-keep-alive- http-check send meth OPTIONS uri /read-only- http-check expect status 200servers:- {name:pg-test-1 ,ip:10.10.10.11 , port:5432 , options:check port 8008 , backup:true}- {name:pg-test-2 ,ip:10.10.10.12 , port:5432 , options:check port 8008 }- {name:pg-test-3 ,ip:10.10.10.13 , port:5432 , options:check port 8008 }
It will be rendered to /etc/haproxy/<service.name>.cfg and take effect after reload.
NODE_EXPORTER
node_exporter_enabled:true# setup node_exporter on this node?node_exporter_port:9100# node exporter listen port, 9100 by defaultnode_exporter_options:'--no-collector.softnet --no-collector.nvme --collector.tcpstat --collector.processes'
node_exporter_enabled
name: node_exporter_enabled, type: bool, level: C
setup node_exporter on this node? default value is true
node_exporter_port
name: node_exporter_port, type: port, level: C
node exporter listen port, 9100 by default
node_exporter_options
name: node_exporter_options, type: arg, level: C
extra server options for node_exporter, default value: --no-collector.softnet --no-collector.nvme --collector.tcpstat --collector.processes
Pigsty enables tcpstat, processes collectors and disable nvme, softnet metrics collectors by default.
PROMTAIL
Promtail will collect logs from other modules, and send them to LOKI
INFRA: Infra logs, collected only on infra nodes.
nginx-access: /var/log/nginx/access.log
nginx-error: /var/log/nginx/error.log
grafana: /var/log/grafana/grafana.log
NODES: Host node logs, collected on all nodes.
syslog: /var/log/messages
dmesg: /var/log/dmesg
cron: /var/log/cron
PGSQL: PostgreSQL logs, collected when a node is defined with pg_cluster.
postgres: /pg/log/postgres/*
patroni: /pg/log/patroni.log
pgbouncer: /pg/log/pgbouncer/pgbouncer.log
pgbackrest: /pg/log/pgbackrest/*.log
REDIS: Redis logs, collected when a node is defined with redis_cluster.
promtail_enabled:true# enable promtail logging collector?promtail_clean:false# purge existing promtail status file during init?promtail_port:9080# promtail listen port, 9080 by defaultpromtail_positions:/var/log/positions.yaml# promtail position status file path
promtail_enabled
name: promtail_enabled, type: bool, level: C
enable promtail logging collector?
default value is true
promtail_clean
name: promtail_clean, type: bool, level: G/A
purge existing promtail status file during init?
default value is false, if you choose to clean, Pigsty will remove the existing state file defined by promtail_positions which means that Promtail will recollect all logs on the current node and send them to Loki again.
promtail_port
name: promtail_port, type: port, level: C
promtail listen port, 9080 by default
default values: 9080
promtail_positions
name: promtail_positions, type: path, level: C
promtail position status file path
default values: /var/log/positions.yaml
Promtail records the consumption offsets of all logs, which are periodically written to the file specified by promtail_positions.
You can also manage tuned profiles with tuned-adm command on nodes:
tuned-adm list # list available profilestuned-adm active # show current profiletuned-adm profile # list active profiletuned-adm profile <name> # switch to profile <name>tuned-adm verify # list active profilecat /var/log/tuned/tuned.log # show tuned log
The node.yml playbook transforms bare computing resources into fully configured, monitored, and service-ready nodes within your Pigsty infrastructure. This comprehensive automation handles everything from basic OS configuration to advanced monitoring setup.
The node-rm.yml playbook performs clean, comprehensive removal of nodes from your Pigsty infrastructure. This automation ensures all services, configurations, and monitoring integrations are properly deregistered and cleaned up, preventing orphaned resources and maintaining system hygiene.
register :remove register from prometheus & nginx- prometheus :remove registered prometheus monitor target- nginx :remove nginx proxy record for haproxy adminvip :remove node keepalived if enabledhaproxy :remove haproxy load balancernode_exporter :remove monitoring exportervip_exporter :remove keepalived_exporter if enabledpromtail :remove loki log agentprofile :remove /etc/profile.d/node.sh
If NTP is not configured, use a public NTP service or sync time with the admin node.
If your nodes already have NTP configured, you can leave it there by setting node_ntp_enabled to false.
Otherwise, if you have Internet access, you can use public NTP services such as pool.ntp.org.
If you don’t have Internet access, at least you can sync time with the admin node with the following:
node_ntp_servers: # NTP servers in /etc/chrony.conf - pool cn.pool.ntp.org iburst
- pool ${admin_ip} iburst # assume non-admin nodes do not have internet access
How to force sync time on nodes?
Use chronyc to sync time. You have to configure the NTP service first.
ansible all -b -a 'chronyc -a makestep'# sync time
You can replace all with any group or host IP address to limit execution scope.
Remote nodes are not accessible via SSH commands.
Consider using Ansible connection parameters if the target machine is hidden behind an SSH springboard machine, or if some customizations have been made that cannot be accessed directly using ssh ip. Additional SSH ports can be specified with ansible_port or ansible_host for SSH Alias.
When performing deployments and changes, the admin user used must have ssh and sudo privileges for all nodes. Password-free is not required.
You can pass in ssh and sudo passwords via the -k|-K parameter when executing the playbook or even use another user to run the playbook via -eansible_host=<another_user>. However, Pigsty strongly recommends configuring SSH passwordless login with passwordless sudo for the admin user.
Create an admin user with the existing admin user.
This will create an admin user specified by node_admin_username using the existing admin user on that node.
Pigsty will try to include all dependencies in the local yum repo on infra nodes. This repo file will be added according to node_repo_modules. And existing repo files will be removed by default according to the default value of node_repo_remove. This will prevent the node from using the Internet repo or some stupid issues.
If you want to keep existing repo files during node init, just set node_repo_remove to false.
If you want to keep existing repo files during infra node local repo bootstrap, just set repo_remove to false.
Why my shell prompt change and how to restore it?
The pigsty prompt is defined with the environment variable PS1 in /etc/profile.d/node.sh.
To restore your existing prompt, just remove that file and login again.
Tencent OpenCloudOS Compatibility Issue
OpenCloudOS does not have softdog module, overwrite node_kernel_modules on global vars:
ETCD is a distributed, reliable key-value store for the most critical data of a distributed system.
etcd is used as DCS (Distributed configuration storage) for patroni, the PostgreSQL HA Agent for config management and leader election.
In short, PGSQL depends on a global ETCD module, and ETCD depends on NODE module to work properly (use the node CA).
You have to define the etcd cluster in the config inventory before deploying it.
Usually you can choose an etcd cluster with:
One Node, no high availability, just the functionality of etcd, suitable for dev, test & demo purpose.
Three Nodes, basic high availability, tolerate one node failure, suitable for medium production env.
Five Nodes, better high availability, tolerate two-node failure, suitable for large production env.
Use even number of etcd nodes is meaningless, and more than five nodes is not common.
One Node
Define the group etcd in the inventory, It will create a singleton etcd instance.
# etcd cluster for ha postgresetcd:{hosts:{10.10.10.10:{etcd_seq:1 } }, vars:{etcd_cluster:etcd } }
This line exists almost in all single-node config template, where the placeholder IP address 10.10.10.10 will be replaced with the current admin node IP.
The only necessary parameters are etcd_seq and etcd_cluster, which uniquely identify the cluster and each instance.
Three Nodes
Three-node etcd cluster is quite common, which tolerates one-node failure, suitable for most cases.
The trio and safe config templates use a three-node etcd cluster, as shown below:
etcd:# dcs service for postgres/patroni ha consensushosts:# 1 node for testing, 3 or 5 for production10.10.10.10:{etcd_seq:1}# etcd_seq required10.10.10.11:{etcd_seq:2}# assign from 1 ~ n10.10.10.12:{etcd_seq:3}# use odd numbersvars:# cluster level parameter override roles/etcdetcd_cluster:etcd # mark etcd cluster name etcdetcd_safeguard:false# safeguard against purging
Five Nodes
Five nodes etcd cluster can tolerate two node failure, suitable for large prod env.
There’s a five-node etcd cluster example in the prod template:
#-----------------------------------------------------------------# ETCD#-----------------------------------------------------------------#etcd_seq: 1 # etcd instance identifier, explicitly requiredetcd_cluster:etcd # etcd cluster & group name, etcd by defaultetcd_data:/data/etcd # etcd data directory, /data/etcd by defaultetcd_learner:false# etcd instance run as learner? false by defaultetcd_port:2379# etcd client port, 2379 by defaultetcd_peer_port:2380# etcd peer port, 2380 by defaultetcd_init:new # etcd initial cluster state, new or existingetcd_election_timeout:1000# etcd election timeout, 1000ms by defaultetcd_heartbeat_interval:100# etcd heartbeat interval, 100ms by default
ETCD_REMOVE Parameters
#-----------------------------------------------------------------# ETCD_REMOVE#-----------------------------------------------------------------etcd_safeguard:false# prevent accidental removal?etcd_rm_data:true# remove etcd data during removal?etcd_rm_pkg:false# uninstall etcd packages during removal?
etcd_seq
name: etcd_seq, type: int, level: I
etcd instance identifier, REQUIRED
no default value, you have to specify it explicitly. Here is a 3-node etcd cluster example:
etcd:# dcs service for postgres/patroni ha consensushosts:# 1 node for testing, 3 or 5 for production10.10.10.10:{etcd_seq:1}# etcd_seq required10.10.10.11:{etcd_seq:2}# assign from 1 ~ n10.10.10.12:{etcd_seq:3}# use odd numbersvars:# cluster level parameter override roles/etcdetcd_cluster:etcd # mark etcd cluster name etcdetcd_safeguard:false# safeguard against purging
etcd_cluster
name: etcd_cluster, type: string, level: C
etcd cluster & group name, etcd by default
default values: etcd, which is a fixed group name, can be useful when you want to use deployed some extra etcd clusters
etcd_learner
name: etcd_learner, type: bool, level: I
Init etcd instance as learner? default value is false
When set to true, the etcd instance will be init as leaner, so it will not be able to vote in the etcd cluster.
You can promote it to a full member later with etcdctl member promote command.
etcd_data
name: etcd_data, type: path, level: C
etcd data directory, /data/etcd by default
etcd_port
name: etcd_port, type: port, level: C
etcd client port, 2379 by default
etcd_peer_port
name: etcd_peer_port, type: port, level: C
etcd peer port, 2380 by default
etcd_init
name: etcd_init, type: enum, level: C
etcd initial cluster state, new or existing
default values: new, which will create a standalone new etcd cluster.
The value existing is used when trying to append new member to existing etcd cluster.
etcd_election_timeout
name: etcd_election_timeout, type: int, level: C
etcd election timeout, 1000 (ms) by default
etcd_heartbeat_interval
name: etcd_heartbeat_interval, type: int, level: C
etcd heartbeat interval, 100 (ms) by default
ETCD_REMOVE
This section contains parameters for the etcd_remove role,
which are action flags for the etcd-rm.yml playbook.
etcd_safeguard
name: etcd_safeguard, type: bool, level: G/C/A
prevent purging etcd instance? default value is false
If enabled, running etcd instance will not be purged by etcd-rm.yml playbook.
etcd_rm_data
name: etcd_rm_data, type: bool, level: G/C/A
remove etcd data during removal? default value is true
When enabled, the etcd-rm.yml playbook will remove etcd data directories and configuration files during cluster or member removal.
etcd_rm_pkg
name: etcd_rm_pkg, type: bool, level: G/C/A
uninstall etcd packages during removal? default value is false
When enabled, the etcd-rm.yml playbook will uninstall etcd packages during cluster or member removal.
If there’s an existing etcd cluster, this playbook will update the config and restart all the etcd instances
Pigsty has a safeguard mechanism to prevent accidental purge. etcd_safeguard is false by default,
For provisioned etcd cluster in prod env, you can enable safeguard to prevent accidental clean.
Architecture Change: Pigsty v3.6+
Since Pigsty v3.6+, the etcd.yml playbook and etcd role are focused solely on cluster installation and member addition. All removal operations have been moved to the dedicated etcd-rm.yml playbook using the etcd_remove role.
Remove Cluster
To remove an existing etcd cluster, you can use the dedicated etcd-rm.yml:
./etcd-rm.yml # remove the default etcd cluster
bin/etcd-rm # remove the default etcd cluster
If the etcd_safeguard is set to true, the playbook will abort.
CLI Environment
Pigsty use etcd v3 API by default. (v2 support is dropped since v3.6.0)
If you want to append a new member to an existing etcd cluster,
you have to add it to the config inventory, and run the playbook with etcd_init = existing
against the new member:
bin/etcd-add <ip> # append new members to existing etcd cluster
It’s usually OK to re-run the playbook, it will update the etcd cluster config and restart etcd instances.
Change Since Pigsty v3.6+
Since Pigsty v3.6+, the etcd.yml playbook no longer has cluster removal capabilities. Use the dedicated etcd-rm.yml playbook and etcd_remove role for etcd cluster removal operations.
This dashboard provides key information about the ETCD status, with the most notable being ETCD Aliveness, which displays the overall service status of the ETCD cluster.
Red bands indicate periods when instances are unavailable, while the blue-gray bands below show when the entire cluster is unavailable.
Alert Rules
Pigsty provides the following two alert rules for the INFRA module:
etcd is a distributed, reliable key-value store used to store the most critical config / consensus data in the deployment.
Pigsty uses etcd as the DCS (Distributed Configuration Store) service for Patroni, which will store the high availability status information of the PostgreSQL cluster.
How many etcd instances should I choose?
If more than (include) half of the etcd instances are down, the etcd cluster, and its service will be unavailable.
For example, a 3-node etcd cluster can tolerate at most one node failure, and the other two nodes can still work normally; while a 5-node etcd cluster can tolerate 2 node failures.
Beware that the learner instances in the etcd cluster do not count in the member number.
So in a 3-node etcd cluster, if there is a learner instance, the actual member count is 2, so no node failure can be tolerated.
It is advisable to choose an odd number of etcd instances to avoid split-brain scenarios. It is recommended to use 3 or 5 nodes for the production environment.
What is the impact of etcd failure?
If etcd cluster is unavailable, it will affect the control plane of Pigsty, but not the data plane — the existing PostgreSQL cluster will continue to serve, but admin operations through Patroni will not work.
During etcd failure, PostgreSQL HA is unable to perform automatic failover; and most of the Patroni operations will be blocked, such as edit-config, restart, switchover, etc…
Admin tasks through Ansible playbooks are usually not affected by etcd failure, such as create database, create user, reload HBA and Service, etc…,
and you can always operate the PostgreSQL cluster directly to achieve most of the patroni functions.
Beware that the above description is only applicable to newer versions of Patroni (>=3.0, Pigsty >= 2.0). If you are using an older version of Patroni (<3.0, corresponding to Pigsty version 1.x), etcd / consul failure will cause a serious impact:
All PostgreSQL clusters will be demoted and reject write requests, and etcd failure will be amplified as a global PostgreSQL failure. After Patroni 3.0’s DCS Failsafe feature, this situation has been significantly improved.
What data is stored in the etcd cluster?
etcd is only used for PostgreSQL HA consensus in Pigsty, no other data is stored in etcd by default.
These consensus data are managed by Patroni, and when these data are lost in etcd, Patroni will automatically rebuild them.
Thus, by default, the etcd in Pigsty can be regarded as a “stateless service” that is disposable, which brings great convenience to maintenance work.
If you use etcd for other purposes, such as storing metadata for Kubernetes, or storing other data, you need to back up the etcd data yourself and restore the data after the etcd cluster is restored.
How to recover from etcd failure?
Since etcd is disposable in Pigsty, you can quickly stop the bleeding by “restarting” or “redeploying” etcd in case of failure.
To Restart the etcd cluster, you can use the following Ansible command (or systemctl restart etcd):
./etcd.yml -t etcd_launch
To Reset the etcd cluster, you can run this playbook, it will nuke the etcd cluster and redeploy it:
./etcd-rm.yml # nuke the existing cluster./etcd.yml # create a new cluster
Beware that if you use etcd to store other data, don’t forget to backup etcd data before nuking the etcd cluster.
Is any maintenance work for etcd cluster?
In short: do not use all the quota of etcd.
etcd has a default quota for database size of 2GB, if your etcd database size exceeds this limit, etcd will reject write requests. Meanwhile, as etcd’s data model illustrates, each write will generate a new version (a.k.a. revision), so if your etcd cluster writes frequently, even with very few keys, the etcd database size may continue to grow, and may fail when it reaches the quota limit.
You can achieve this by Auto Compact, Manual Compact, Defragmentation, and Quota Increase, etc., please refer to the etcd official maintenance guide.
Pigsty has auto compact enabled by default since v2.6, so you usually don’t have to worry about etcd full. For versions before v2.6, we strongly recommend enabling etcd’s auto compact feature in the production environment.
Fill etcd may lead to PostgreSQL failure!
For Pigsty v2.0 - v2.5 users, we strongly recommend upgrading to a newer version, or following the instructions below to enable etcd auto compaction!
How to enable etcd auto compaction?
If you are using an earlier version of Pigsty (v2.0 - v2.5), we strongly recommend that you enable etcd’s auto compaction feature in the production environment.
You can set all the PostgreSQL cluster to maintenance mode and then redeploy the etcd cluster with ./etcd.yml to apply the these changes.
It will increase the etcd default quota from 2 GiB to 16 GiB, and ensure that only the most recent day’s write history is retained, avoiding the infinite growth of the etcd database size.
Where does the PostgreSQL HA data store in etcd?
Patroni will use the pg_namespace (default is /pg) as the prefix for all metadata keys in etcd, followed by the PostgreSQL cluster name.
For example, a PG cluster named pg-meta, its metadata keys will be stored under /pg/pg-meta, which may look like this:
The hard-coded group, etcd, will be used as DCS servers for PGSQL. You can initialize them with etcd.yml or assume it is an existing external etcd cluster.
To use an existing external etcd cluster, define them as usual and make sure your current etcd cluster certificate is signed by the same CA as your self-signed CA for PGSQL.
How to add a new member to the existing etcd cluster?
etcdctl member remove <etcd_server_id> # kick member out of the cluster (on admin node)./etcd.yml -l <ins_ip> -t etcd_purge # purge etcd instance
15 - MinIO
The open source object storage service
Min.IO: S3-Compatible Open-Source Multi-Cloud Object Storage designed to be scalable, secure, and handy.
It has native multi-node multi-driver HA support and can store documents, pictures, videos, and backups. It’s an optional module in Pigsty.
You can use MinIO as an optional PostgreSQL backup storage repo, in addition to the default local posix FS repo. If the MinIO repo is used,
the MINIO module should be installed before any PGSQL modules. MinIO requires a trusted CA to work, so you have to install it after NODE.
Configuration
Configure minio module, and use multiple minio nodes.
Get started with MinIO and MCli, how to access the MinIO service?
After MinIO cluster is configured and deployed with the playbook, you can start using and accessing the MinIO cluster by following the instructions here.
Deploy Cluster
It is straightforward to deploy a single-node MinIO instance with Pigsty.
The install.yml playbook will automatically create the MinIO cluster defined in the inventory, so you don’t need to run the minio.yml playbook manually, if you choose the default one-pass installation.
If you plan to deploy a production-grade large-scale multi-node MinIO cluster, we strongly recommend you to read the Pigsty MinIO configuration document and the MinIO document before proceeding.
Access Cluster
You have to access MinIO via HTTPS, so make sure the default minio service domain (sss.pigsty) point to the right place:
You can add static resolution records in node_etc_hosts or manually modify the /etc/hosts file
You can add a record on the internal DNS server if you are using DNS service
You can add a record in dns_records if you are using the DNSMASQ on infra nodes
It is recommended to use the first method: static DNS resolution records to avoid MinIO’s additional dependency on DNS in production environments.
You have to point the MinIO service domain to the IP address and service port of the MinIO server node, or the IP address and service port of the load balancer. Pigsty will use the default domain name sss.pigsty and default port 9000.
For example, if you are using haproxy to expose MinIO service like this, the port may be 9002.
Adding Alias
To access the MinIO server cluster using the mcli client, you need to configure the server alias first:
mcli alias ls # list minio alias (the default is sss)mcli aliasset sss https://sss.pigsty:9000 minioadmin minioadmin # root usermcli aliasset sss https://sss.pigsty:9002 minioadmin minioadmin # root user, on load balancer port 9002mcli aliasset pgbackrest https://sss.pigsty:9000 pgbackrest S3User.Backup # use another user
There’s a pre-configured MinIO alias named sss on the admin user of the admin node, you can use it directly.
For the full functionality of the MinIO client tool mcli, please refer to the documentation: MinIO Client.
Manage User
You can manage biz users in MinIO using mcli, for example, you can create the two default biz users using the command line:
mcli admin user list sss # list all usersset +o history# hide shell historymcli admin user add sss dba S3User.DBA
mcli admin user add sss pgbackrest S3User.Backup
set -o history
Manage Bucket
You can manage bucket with mcli:
mcli ls sss/ # list all bucket on 'sss'mcli mb --ignore-existing sss/hello # create a bucket named 'hello'mcli rb --force sss/hello # delete the 'hello' bucket
Mange Object
You can perform object CRUD with cli, for example:
mcli cp /www/pigsty/* sss/infra/ # upload local repo content to infra bucketmcli cp sss/infra/plugins.tgz /tmp/ # download file to local from miniomcli ls sss/infra # list all files in the infra bucketmcli rm sss/infra/plugins.tgz # delete file in infra bucketmcli cat sss/infra/repo_complete # output the content of
Pigsty repo has rclone available, a convenient cloud object storage client that you can use to access MinIO services.
yum install rclone;# el compatiblednf install rclone;# debian/ubuntumkdir -p ~/.config/rclone/;tee ~/.config/rclone/rclone.conf > /dev/null <<EOF
[sss]
type = s3
access_key_id = minioadmin
secret_access_key = minioadmin
endpoint = sss.pigsty:9000
EOFrclone ls sss:/
Backup Repo
The MinIO is used as a backup repository for pgBackRest by default in Pigsty. When you modify the pgbackrest_method to minio, the PGSQL module will automatically switch the backup repository to MinIO.
pgbackrest_method:local # pgbackrest repo method: local,minio,[user-defined...]pgbackrest_repo:# pgbackrest repo: https://pgbackrest.org/configuration.html#section-repositorylocal:# default pgbackrest repo with local posix fspath:/pg/backup # local backup directory, `/pg/backup` by defaultretention_full_type:count # retention full backups by countretention_full:2# keep 2, at most 3 full backup when using local fs repominio:# optional minio repo for pgbackresttype:s3 # minio is s3-compatible, so s3 is useds3_endpoint:sss.pigsty # minio endpoint domain name, `sss.pigsty` by defaults3_region:us-east-1 # minio region, us-east-1 by default, useless for minios3_bucket:pgsql # minio bucket name, `pgsql` by defaults3_key:pgbackrest # minio user access key for pgbackrests3_key_secret:S3User.Backup # minio user secret key for pgbackrests3_uri_style:path # use path style uri for minio rather than host stylepath:/pgbackrest # minio backup path, default is `/pgbackrest`storage_port:9000# minio port, 9000 by defaultstorage_ca_file:/pg/cert/ca.crt # minio ca file path, `/pg/cert/ca.crt` by defaultbundle:y# bundle small files into a single filecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest # AES encryption password, default is 'pgBackRest'retention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for last 14 days
Beware that if you are using MinIO through load balancer, you should use the corresponding domain name and port number here.
15.2 - Configure
Define SNSD, SNMD, MNMD MinIO Clusters
You have to define a MinIO cluster in the config inventory before deploying it.
There are 3 major deployment modes for MinIO clusters:
SNMD: Single-Node Multi-Drive: a compromise mode, use multiple disks (>=2) on a single server, only when resources are extremely limited.
MNMD: Multi-Node Multi-Drive: standard production deployment with the best reliability, but requires multiple servers and real drivers.
We recommend using SNSD and MNMD for development and production deployment, respectively, and SNMD only when resources are extremely limited (only one server).
When using a multi-node MinIO cluster, you can access the service from any node, so the best practice is to use a load balancer and HA access.
Core Param
There’s one and only one core param for MinIO deployment, which is MINIO_VOLUMES, which specifies the nodes, drivers, pools of a minio cluster
Pigsty will auto-generate MINIO_VOLUMES according to the config inventory for you, but you can always override it directly. If not explicitly specified, Pigsty will generate it according to the following rules:
SNSD: MINIO_VOLUMES points to any dir on local node, from minio_data
SNMD: MINIO_VOLUMES points to a series of real drivers on local node, from minio_data
MNMD: MINIO_VOLUMES points to multiple nodes & multiple drivers, according to minio_data and minio_node
Use minio_data to specify drivers on each node, such as /data{1...4}
Use minio_node to specify node name pattern, such as ${minio_cluster}-${minio_seq}.pigsty
Multi-Pool: MINIO_VOLUMES need to be explicitly specified
The only required params are minio_seq and minio_cluster, which generate a unique identity for each MinIO instance.
Single-Node Single-Driver mode is for dev purposes, so you can use a common dir as the data dir. The default data dir for SNSD minio is specified by minio_data, which is /data/minio by default. Beware that in multi-driver or multi-node mode, MinIO will refuse to start if using a common dir as the data dir rather than a mount point.
We strongly recommend using a static domain name record to access MinIO. For example, the default sss.pigsty if minio_domain can be added to all nodes through:
node_etc_hosts:["10.10.10.10 sss.pigsty"]# domain name to access minio from all nodes (required)
To use multiple disks on a single node, you have to specify the minio_data in the format of {{ prefix }}{x...y}, which defines a series of disk mount points.
minio:hosts:{10.10.10.10:{minio_seq:1}}vars:minio_cluster:minio # minio cluster name, minio by defaultminio_data:'/data{1...4}'# minio data dir(s), use {x...y} to specify multi drivers
Use real drivers and mountpoint
Beware that in multi-driver or multi-node mode, MinIO will refuse to start if using a common dir as the data dir rather than a mount point.
This example defines a single-node MinIO cluster with 4 drivers: /data1, /data2, /data3, /data4. You have to mount them properly before launching MinIO:
The vagrant MinIO sandbox has a pre-defined 4-node MinIO cluster with 4 drivers. You have to properly mount them before starting MinIO (be sure to format disks with xfs):
mkfs.xfs /dev/vdb; mkdir /data1; mount -t xfs /dev/sdb /data1;mkfs.xfs /dev/vdc; mkdir /data2; mount -t xfs /dev/sdb /data2;mkfs.xfs /dev/vdd; mkdir /data3; mount -t xfs /dev/sdb /data3;mkfs.xfs /dev/vde; mkdir /data4; mount -t xfs /dev/sdb /data4;
Disk management is beyond this topic, just make sure your /etc/fstab is properly configured to auto-mount disks after reboot.
SNMD mode can utilize multiple disks on a single server to provide higher performance and capacity, and tolerate partial disk failures.
But it can do nothing with node failure, and you can’t add new nodes at runtime, so we don’t recommend using SNMD mode in production unless you have a special reason.
The extra minio_node param will be used for a multi-node deployment in addition to the minio_data
For example, this configuration defines a 4-node MinIO cluster with 4 drivers per node:
minio:hosts:10.10.10.10:{minio_seq:1}# nodename: minio-1.pigsty10.10.10.11:{minio_seq:2}# nodename: minio-2.pigsty10.10.10.12:{minio_seq:3}# nodename: minio-3.pigsty10.10.10.13:{minio_seq:4}# nodename: minio-4.pigstyvars:minio_cluster:miniominio_data:'/data{1...4}'# 4-disk per nodeminio_node:'${minio_cluster}-${minio_seq}.pigsty'# minio name pattern
The minio_node param specifies the MinIO node name pattern, which is ${minio_cluster}-${minio_seq}.pigsty by default. The server name is very important for MinIO to identify and access other nodes in the cluster. It will be populated with minio_cluster and minio_seq, and write to /etc/hosts of all minio cluster members.
In this case, the MINIO_VOLUMES will be set to https://minio-{1...4}.pigsty/data{1...4} to identify the 16 disks on 4 nodes.
Multi-Pool
MinIO’s architecture allows for cluster expansion by adding new storage pools. In Pigsty, you can achieve this by explicitly specifying the minio_volumes param to specify nodes/disks for each pool.
For example, suppose you have already created a MinIO cluster as defined in the Multi-Node Multi-Disk example, and now you want to add a new storage pool consisting of four nodes.
You can specify minio_volumes here to allocate nodes for each pool to scale out the cluster.
Please note that by default, Pigsty allows only one MinIO cluster per deployment. If you need to deploy multiple MinIO clusters, some parameters with default values need to be explicitly set and cannot be omitted to avoid naming conflicts, as shown above.
Expose Service
MinIO will serve on port 9000 by default. If a multi-node MinIO cluster is deployed, you can access its service via any node. It would be better to expose MinIO service via a load balancer, such as the default haproxy on NODE, or use the L2 vip.
To expose MinIO service with haproxy, you have to define an extra service with haproxy_services:
minio:hosts:10.10.10.10:{minio_seq:1 , nodename:minio-1 }10.10.10.11:{minio_seq:2 , nodename:minio-2 }10.10.10.12:{minio_seq:3 , nodename:minio-3 }vars:minio_cluster:minionode_cluster:miniominio_data:'/data{1...2}'# use two disk per nodeminio_node:'${minio_cluster}-${minio_seq}.pigsty'# minio node name patternhaproxy_services:# EXPOSING MINIO SERVICE WITH HAPROXY- name:minio # [REQUIRED] service name, uniqueport:9002# [REQUIRED] service port, uniqueoptions:# [OPTIONAL] minio health check- option httpchk- option http-keep-alive- http-check send meth OPTIONS uri /docs/minio/health/live- http-check expect status 200servers:- {name:minio-1 ,ip:10.10.10.10 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-2 ,ip:10.10.10.11 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-3 ,ip:10.10.10.12 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}
MinIO uses port 9000 by default. A multi-node MinIO cluster can be accessed by connecting to any one of its nodes.
Service access falls under the scope of the NODE module, and we’ll provide only a basic introduction here.
High-availability access to a multi-node MinIO cluster can be achieved using an L2 VIP or HAProxy. For example, you can use Keepalived to bind an L2 VIP to the MinIO cluster,
or use the haproxy component provided by the NODE module to expose MinIO services through a load balancer.
# minio cluster with 4 nodes and 4 drivers per nodeminio:hosts:10.10.10.10:{minio_seq:1 , nodename:minio-1 }10.10.10.11:{minio_seq:2 , nodename:minio-2 }10.10.10.12:{minio_seq:3 , nodename:minio-3 }10.10.10.13:{minio_seq:4 , nodename:minio-4 }vars:minio_cluster:miniominio_data:'/data{1...4}'minio_buckets:[{name:pgsql }, { name: infra }, { name: redis } ]minio_users:- {access_key:dba , secret_key:S3User.DBA, policy:consoleAdmin }- {access_key:pgbackrest , secret_key:S3User.SomeNewPassWord , policy:readwrite }# bind a node l2 vip (10.10.10.9) to minio cluster (optional)node_cluster:miniovip_enabled:truevip_vrid:128vip_address:10.10.10.9vip_interface:eth1# expose minio service with haproxy on all nodeshaproxy_services:- name:minio # [REQUIRED] service name, uniqueport:9002# [REQUIRED] service port, uniquebalance:leastconn # [OPTIONAL] load balancer algorithmoptions:# [OPTIONAL] minio health check- option httpchk- option http-keep-alive- http-check send meth OPTIONS uri /docs/minio/health/live- http-check expect status 200servers:- {name:minio-1 ,ip:10.10.10.10 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-2 ,ip:10.10.10.11 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-3 ,ip:10.10.10.12 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-4 ,ip:10.10.10.13 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}
In the configuration above, HAProxy is enabled on all nodes of the MinIO cluster, exposing MinIO services on port 9002, and a Layer 2 VIP is bound to the cluster.
When in use, users should point the sss.pigsty domain name to the VIP address 10.10.10.9 and access MinIO services using port 9002.
This ensures high availability, as the VIP will automatically switch to another node if any node fails.
In this scenario, you may also need to globally modify the destination of domain name resolution
and adjust the minio_endpoint parameter to change the endpoint address corresponding to the MinIO alias on the management node:
minio_endpoint:https://sss.pigsty:9002 # Override the default https://sss.pigsty:9000node_etc_hosts:["10.10.10.9 sss.pigsty"]# Other nodes will use the sss.pigsty domain
Dedicate Proxies
Pigsty allows using dedicate load balancer cluster instead of the node cluster itself to run VIP & HAProxy.
proxy:hosts:10.10.10.18 :{nodename:proxy1 ,node_cluster:proxy ,vip_interface:eth1 ,vip_role:master }10.10.10.19 :{nodename:proxy2 ,node_cluster:proxy ,vip_interface:eth1 ,vip_role:backup }vars:vip_enabled:truevip_address:10.10.10.20vip_vrid:20haproxy_services:# expose minio service : sss.pigsty:9000- name:minio # [REQUIRED] service name, uniqueport:9000# [REQUIRED] service port, uniquebalance:leastconn# Use leastconn algorithm and minio health checkoptions:["option httpchk","option http-keep-alive","http-check send meth OPTIONS uri /docs/minio/health/live","http-check expect status 200"]servers:# reload service with ./node.yml -t haproxy_config,haproxy_reload- {name:minio-1 ,ip:10.10.10.21 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-2 ,ip:10.10.10.22 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-3 ,ip:10.10.10.23 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-4 ,ip:10.10.10.24 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-5 ,ip:10.10.10.25 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}
In this case, you need to manually configure the DNS resolution to point sss.pigsty to the VIP address of dedicated proxies cluster
minio_endpoint:https://sss.pigsty:9002 # overwrite the defaults: https://sss.pigsty:9000node_etc_hosts:["10.10.10.20 sss.pigsty"]# domain name to access minio from all nodes (required)
# This is the newly added HA MinIO Repo definition, USE THIS INSTEAD!minio_ha:type:s3s3_endpoint:minio-1.pigsty # s3_endpoint could be any load balancer: 10.10.10.1{0,1,2}, or domain names point to any of the 3 nodess3_region:us-east-1 # you could use external domain name: `sss.pigsty` ,which resolve to any members (`minio_domain`)s3_bucket:pgsql # instance & nodename can be used : minio-1.pigsty minio-1.pigsty minio-1.pigsty minio-1 minio-2 minio-3s3_key:pgbackrest # Betters using a new password for MinIO pgbackrest users3_key_secret:S3User.SomeNewPassWords3_uri_style:pathpath:/pgbackreststorage_port:9002# Use the load balancer port 9002 instead of default 9000 (direct access)storage_ca_file:/etc/pki/ca.crtbundle:ycipher_type:aes-256-cbc # Better using a new cipher password for your production environmentcipher_pass:pgBackRest.With.Some.Extra.PassWord.And.Salt.${pg_cluster}retention_full_type:timeretention_full:14
Expose Console
MinIO has a built-in console that can be accessed via HTTPS @ minio_admin_port. If you want to expose the MinIO console to the outside world, you can add MinIO to infra_portal.
Beware that MinIO console should be accessed via HTTPS, please DO NOT expose MinIO console without encryption in production.
Which means you usually need to add m.pigsty resolution to your DNS server, or /etc/hosts on your local host, to access the MinIO console.
Meanwhile, if you are using Pigsty’s self-signed CA rather than a regular public CA,
you usually need to manually trust the CA or certificate to skip the “insecure” warning in the browser.
15.3 - Parameter
Customize MinIO parameters
MinIO is a S3 compatible object storage service. Which is used as an optional central backup storage repo for PostgreSQL.
You may also use it for other purposes, such as storing large files, documents, pictures & videos.
Parameters
There are 21 documented settings: 16 declared MINIO parameters, 2 derived values that may be overridden, and 3 MINIO_REMOVE flags.
#-----------------------------------------------------------------# MINIO#-----------------------------------------------------------------#minio_seq: 1 # minio instance identifier, REQUIREDminio_cluster:minio # minio cluster identifier, REQUIREDminio_user:minio # minio os user, `minio` by defaultminio_https:true# use https for minio, true by defaultminio_node:'${minio_cluster}-${minio_seq}.pigsty'# minio node name patternminio_data:'/data/minio'# minio data dir(s), use {x...y} to specify multi drivers#minio_volumes: # minio data volumes, override defaults if specifiedminio_domain:sss.pigsty # minio external domain name, `sss.pigsty` by defaultminio_port:9000# minio service port, 9000 by defaultminio_admin_port:9001# minio console port, 9001 by defaultminio_access_key:minioadmin # root access key, `minioadmin` by defaultminio_secret_key:minioadmin # root secret key, `minioadmin` by defaultminio_extra_vars:''# extra environment variablesminio_provision:true# run minio provisioning tasks?minio_alias:sss # alias name for local minio deployment#minio_endpoint: https://sss.pigsty:9000 # if not specified, overwritten by defaultsminio_buckets:# list of minio bucket to be created- {name:pgsql }- {name:meta ,versioning:true}- {name:data }minio_users:# list of minio user to be created- {access_key:pgbackrest ,secret_key:S3User.Backup ,policy:pgsql }- {access_key:s3user_meta ,secret_key:S3User.Meta ,policy:meta }- {access_key:s3user_data ,secret_key:S3User.Data ,policy:data }
#-----------------------------------------------------------------# MINIO_REMOVE#-----------------------------------------------------------------minio_safeguard:false# prevent accidental removal?minio_rm_data:true# remove minio data during removal?minio_rm_pkg:false# uninstall minio packages during removal?
minio_seq
name: minio_seq, type: int, level: I
minio instance identifier, REQUIRED identity parameters. no default value, you have to assign it manually
minio_cluster
name: minio_cluster, type: string, level: C
minio cluster name, minio by default. This is useful when deploying multiple MinIO clusters
minio_user
name: minio_user, type: username, level: C
minio os user name, minio by default
minio_https
name: minio_https, type: bool, level: G
Use HTTPS or HTTP for MinIO service, true by default, means using HTTPS.
Beware that pgbackrest requires MinIO HTTPS to work properly, but if you don’t use minio for that, and don’t want to use HTTPS for MinIO, you can set this to false.
minio_node
name: minio_node, type: string, level: C
minio node name pattern, this is used for multi-node deployment
In case of SNSD or SNMD deployment, minio_volumes directly uses the value of minio_data
In case of MNMD deployment, minio_volumes uses the values of minio_node, minio_port, minio_data to generate this param:
In case of multiple storage pool, you have to override minio_volumes to specify multiple node pools explicitly.
It user’s responsibility to make sure the parameters used in minio_volumes are consistent with minio_node, minio_port, minio_data.
minio_domain
name: minio_domain, type: string, level: G
minio service domain name, sss.pigsty by default.
The client can access minio S3 service via this domain name. This name will be registered to local DNSMASQ and included in SSL certs.
minio_port
name: minio_port, type: port, level: C
minio service port, 9000 by default
minio_admin_port
name: minio_admin_port, type: port, level: C
minio console port, 9001 by default
minio_access_key
name: minio_access_key, type: username, level: C
root access key, minioadmin by default
minio_secret_key
name: minio_secret_key, type: password, level: C
root secret key, minioadmin by default
default values: minioadmin
Change PASSWORDS!
It’s very important to change this password in your deployment!
minio_extra_vars
name: minio_extra_vars, type: string, level: C
extra environment variables for minio server. Check Minio Server for the complete list.
default value is empty string, you can use multiline string to passing multiple environment variables.
minio_alias
name: minio_alias, type: string, level: G
MinIO alias name for the local MinIO cluster
default values: sss, which will be written to infra nodes’ / admin users’ client alias profile.
minio_endpoint
name: minio_endpoint, type: string, level: C
The corresponding host:port for the above MinIO alias. This parameter is not defined by default.
If not defined, it will be overwritten by the following default value:
mcli aliasset{{ minio_alias }}{% if minio_endpoint is defined and minio_endpoint !='' %}{{ minio_endpoint }}{% else %}https://{{ minio_domain }}:{{ minio_port }}{% endif %}{{ minio_access_key }}{{ minio_secret_key }}
This alias & endpoint will be added to the admin user on the admin node.
minio_buckets
name: minio_buckets, type: bucket[], level: C
list of minio bucket to be created by default:
minio_buckets:# list of minio bucket to be created- {name:pgsql }- {name:meta ,versioning:true}- {name:data }
Three default buckets are created by default, with different policies.
The pgsql bucket is used for PostgreSQL backups by default. while meta and data are open buckets for other purposes.
For example, the supabase template may use the data bucket to store business data.
And if you have important metadata that requires versioning, you can use the meta bucket out-of-the-box.
Every bucket will have a corresponding policy, with names same as the bucket name. e.g., the pgsql policy have all the privileges on the pgsql bucket, and so on.
You can also add the lock flag to bucket definition, which will enable the object locking feature to prevent accidental deletion of objects in the bucket.
minio_users
name: minio_users, type: user[], level: C
list of minio users to be created, default value:
minio_users:# list of minio user to be created- {access_key:pgbackrest ,secret_key:S3User.Backup ,policy:pgsql }- {access_key:s3user_meta ,secret_key:S3User.Meta ,policy:meta }- {access_key:s3user_data ,secret_key:S3User.Data ,policy:data }
Two default users are created for PostgreSQL DBA and pgBackREST.
Change PASSWORDS!
Please change these passwords in serious production deployments.
minio_safeguard
name: minio_safeguard, type: bool, level: G/C/A
prevent accidental removal? default value is false
If enabled, the minio-rm.yml playbook will abort and refuse to remove the MinIO cluster, providing protection against accidental deletions.
minio_rm_data
name: minio_rm_data, type: bool, level: G/C/A
remove minio data during removal? default value is true
When enabled, the minio-rm.yml playbook will remove MinIO data directories and configuration files during cluster removal.
minio_rm_pkg
name: minio_rm_pkg, type: bool, level: G/C/A
uninstall minio packages during removal? default value is false
When enabled, the minio-rm.yml playbook will uninstall MinIO packages during cluster removal. This is disabled by default to preserve the MinIO installation for potential future use.
The minio_cluster param mark this cluster as a MinIO cluster, and the minio_seq is the sequence number of the MinIO node, which is used to generate MinIO node name like minio-1, minio-2, etc.
This snippet defines a single-node MinIO cluster, using the following command to create the MinIO cluster:
./minio.yml -l minio # init MinIO module on the minio group
Remove Cluster
To destroy an existing MinIO cluster, use the dedicated minio-rm.yml playbook:
./minio-rm.yml -l minio # Remove MinIO cluster
You can also customize the removal process with parameters:
Since Pigsty v3.6+, MinIO cluster removal has been moved to the dedicated minio-rm.yml playbook using the minio_remove role. The prometheus monitoring targets are automatically cleaned up during the removal process.
You cannot scale MinIO at node/disk level, but you can scale at storage pool (multiple nodes) level.
Assume you have a 4-node MinIO cluster and want to double the capacity by adding another four-node storage pool.
minio:hosts:10.10.10.10:{minio_seq:1 , nodename:minio-1 }10.10.10.11:{minio_seq:2 , nodename:minio-2 }10.10.10.12:{minio_seq:3 , nodename:minio-3 }10.10.10.13:{minio_seq:4 , nodename:minio-4 }vars:minio_cluster:miniominio_data:'/data{1...4}'minio_buckets:[{name:pgsql }, { name: infra }, { name: redis } ]minio_users:- {access_key:dba , secret_key:S3User.DBA, policy:consoleAdmin }- {access_key:pgbackrest , secret_key:S3User.SomeNewPassWord , policy:readwrite }# bind a node l2 vip (10.10.10.9) to minio cluster (optional)node_cluster:miniovip_enabled:truevip_vrid:128vip_address:10.10.10.9vip_interface:eth1# expose minio service with haproxy on all nodeshaproxy_services:- name:minio # [REQUIRED] service name, uniqueport:9002# [REQUIRED] service port, uniquebalance:leastconn # [OPTIONAL] load balancer algorithmoptions:# [OPTIONAL] minio health check- option httpchk- option http-keep-alive- http-check send meth OPTIONS uri /docs/minio/health/live- http-check expect status 200servers:- {name:minio-1 ,ip:10.10.10.10 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-2 ,ip:10.10.10.11 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-3 ,ip:10.10.10.12 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}- {name:minio-4 ,ip:10.10.10.13 ,port:9000 ,options:'check-ssl ca-file /etc/pki/ca.crt check port 9000'}
Step 1, add 4 node definitions in the group, allocate sequence number 5 to 8.
The key step is to modify the minio_volumes param, assign the new 4 nodes to a new storage pool.
MinIO cannot scale down at the node/disk level, but you can retire at the storage pool (multiple nodes) level —— Add a new storage pool, drain the old storage pool, migrate to the new storage pool, and then retire the old storage pool.
# 1. remove failure nodebin/node-rm <your_old_node_ip>
# 2. replace failure node with the same name (modify the inventory in case of IP change)bin/node-add <your_new_node_ip>
# 3. provisioning MinIO on new node./minio.yml -l <your_new_node_ip>
# 4. instruct MinIO to perform heal actionmc admin heal
Since Pigsty v3.6+, the minio.yml playbook and minio role are focused solely on cluster installation. All removal operations have been moved to the dedicated minio-rm.yml playbook using the minio_remove role.
Trusted ca file: /etc/pki/ca.crt should exist on all nodes already. which is generated in role: ca and loaded & trusted by default in role: node.
You should install MINIO module on Pigsty-managed nodes (i.e., Install NODE first)
Configure redis module, and use multiple redis nodes.
Parameters
Customize redis components with 21 parameters
Administration
Create, remove, expand, redis cluster
Playbooks
Ansible playbooks that can be used in this module
Monitoring
Dashboards, metrics, record & alerting rules.
FAQ
Frequently Asked Questions about redis module
16.1 - Configure
describe the redis cluster you want
The entity model of Redis is almost the same as that of PostgreSQL,
which also includes the concepts of Cluster and Instance. The Cluster here does not refer to the native Redis Cluster mode.
The core difference between the REDIS module and the PGSQL module is that Redis uses a single-node multi-instance deployment rather than the 1:1 deployment:
multiple Redis instances are typically deployed on a physical/virtual machine node to utilize multicore CPUs fully.
Therefore, the ways to configure and administer Redis instances are slightly different from PGSQL.
In Redis managed by Pigsty, nodes are entirely subordinate to the cluster, which means that currently,
it is not allowed to deploy Redis instances of two different clusters on one node.
However, this does not affect deploying multiple independent Redis primary replica instances on one node.
Redis Identity
Redis identity parameters are required parameters when defining a Redis cluster.
standalone: setup Redis in standalone (master-slave) mode
cluster: setup this Redis cluster as a Redis native cluster
sentinel: setup Redis as a sentinel for standalone Redis HA
Here are three examples:
A 1-node, one master & one slave Redis Standalone cluster: redis-ms
A 1-node, 3-instance Redis Sentinel cluster: redis-sentinel
A 2-node, 6-instance Redis Cluster: redis-cluster
redis-ms:# redis classic primary & replicahosts:{10.10.10.10:{redis_node:1 , redis_instances:{6379:{}, 6380:{replica_of:'10.10.10.10 6379'}}}}vars:{redis_cluster:redis-ms ,redis_password:'redis.ms' ,redis_max_memory:64MB }redis-meta:# redis sentinel x 3hosts:{10.10.10.11:{redis_node:1 , redis_instances:{26379:{} ,26380:{} ,26381:{}}}}vars:redis_cluster:redis-metaredis_password:'redis.meta'redis_mode:sentinelredis_max_memory:16MBredis_sentinel_monitor:# primary list for redis sentinel, use cls as name, primary ip:port- {name:redis-ms, host:10.10.10.10, port:6379 ,password:redis.ms, quorum:2}redis-test:# redis native cluster: 3m x 3shosts:10.10.10.12:{redis_node:1 ,redis_instances:{6379:{} ,6380:{} ,6381:{}}}10.10.10.13:{redis_node:2 ,redis_instances:{6379:{} ,6380:{} ,6381:{}}}vars:{redis_cluster:redis-test ,redis_password:'redis.test' ,redis_mode:cluster, redis_max_memory:32MB }
Limitation
A Redis node can only belong to one Redis cluster, which means you cannot assign a node to two different Redis clusters simultaneously.
On each Redis node, you need to assign a unique port number to the Redis instance to avoid port conflicts.
Typically, the same Redis cluster will use the same password, but multiple Redis instances on a Redis node cannot set different passwords (because redis_exporter only allows one password).
Redis Cluster has built-in HA, while standalone HA requires manually configured in Sentinel because we are unsure if you have any sentinels available.
Fortunately, configuring standalone Redis HA is straightforward: Configure HA with sentinel.
#redis_cluster: <CLUSTER> # redis cluster name, required identity parameter#redis_node: 1 <NODE> # redis node sequence number, node int id required#redis_instances: {} <NODE> # redis instances definition on this redis noderedis_fs_main:/data # redis main data mountpoint, `/data` by defaultredis_exporter_enabled:true# install redis exporter on redis nodes?redis_exporter_port:9121# redis exporter listen port, 9121 by defaultredis_exporter_options:''# cli args and extra options for redis exporterredis_safeguard:false# prevent purging running redis instance?redis_clean:true# purging existing redis during init?redis_rmdata:true# remove redis data when purging redis server?redis_mode:standalone # redis mode: standalone,cluster,sentinelredis_conf:redis.conf # redis config template path, except sentinelredis_bind_address:'0.0.0.0'# redis bind address, empty string will use host ipredis_max_memory:1GB # max memory used by each redis instanceredis_mem_policy:allkeys-lru # redis memory eviction policyredis_password:''# redis password, empty string will disable passwordredis_rdb_save:['1200 1']# redis rdb save directives, disable with empty listredis_aof_enabled:false# enable redis append only file?redis_rename_commands:{}# rename redis dangerous commandsredis_cluster_replicas:1# replica number for one master in redis clusterredis_sentinel_monitor:[]# sentinel master list, works on sentinel cluster only
redis_cluster
name: redis_cluster, type: string, level: C
redis cluster name, required identity parameter.
no default value, you have to define it explicitly.
Comply with regexp [a-z][a-z0-9-]*, it is recommended to use the same name as the group name and start with redis-
redis_node
name: redis_node, type: int, level: I
redis node sequence number, unique integer among redis cluster is required
You have to explicitly define the node id for each redis node. integer start from 0 or 1.
redis_instances
name: redis_instances, type: dict, level: I
redis instances definition on this redis node
no default value, you have to define redis instances on each redis node using this parameter explicitly.
Here is an example for a native redis cluster definition
redis bind address, empty string will use inventory hostname
default values: 0.0.0.0, which will bind to all available IPv4 address on this host
PLEASE bind to intranet IP only in production environment, i.e. set this value to ''
redis_max_memory
name: redis_max_memory, type: size, level: C/I
max memory used by each redis instance, default values: 1GB
redis_mem_policy
name: redis_mem_policy, type: enum, level: C
redis memory eviction policy
default values: allkeys-lru, check redis eviction policy for more details
noeviction: New values aren’t saved when memory limit is reached. When a database uses replication, this applies to the primary database
allkeys-lru: Keeps most recently used keys; removes least recently used (LRU) keys
allkeys-lfu: Keeps frequently used keys; removes least frequently used (LFU) keys
volatile-lru: Removes least recently used keys with the expire field set to true.
volatile-lfu: Removes least frequently used keys with the expire field set to true.
allkeys-random: Randomly removes keys to make space for the new data added.
volatile-random: Randomly removes keys with expire field set to true.
volatile-ttl: Removes keys with expire field set to true and the shortest remaining time-to-live (TTL) value.
redis_password
name: redis_password, type: password, level: C/N
redis password, empty string will disable password, which is the default behavior
Note that due to the implementation limitation of redis_exporter, you can only set one redis_password per node. This is usually not a problem, because pigsty does not allow deploying two different redis cluster on the same node.
PLEASE use a strong password in production environment
redis_rdb_save
name: redis_rdb_save, type: string[], level: C
redis rdb save directives, disable with empty list, check redis persist for details.
the default value is ["1200 1"]: dump the dataset to disk every 20 minutes if at least 1 key changed:
redis_aof_enabled
name: redis_aof_enabled, type: bool, level: C
enable redis append only file? default value is false.
redis_rename_commands
name: redis_rename_commands, type: dict, level: C
rename redis dangerous commands, which is a dict of k:v old: new
default values: {}, you can hide dangerous commands like FLUSHDB and FLUSHALL by setting this value, here’s an example:
replica number for one master/primary in redis cluster, default values: 1
redis_sentinel_monitor
name: redis_sentinel_monitor, type: master[], level: C
This can only be used when redis_mode is set to sentinel.
List of redis master to be monitored by this sentinel cluster. each master is defined as a dict with name, host, port, password, quorum keys.
redis_sentinel_monitor:# primary list for redis sentinel, use cls as name, primary ip:port- {name:redis-src, host:10.10.10.45, port:6379 ,password:redis.src, quorum:1}- {name:redis-dst, host:10.10.10.48, port:6379 ,password:redis.dst, quorum:1}
The name and host are mandatory, port, password, quorum are optional, quorum is used to set the quorum for this master, usually large than half of the sentinel instances.
16.3 - Administration
run administrative tasks
Here are some common administration tasks for Redis. Check FAQ: Redis for more details.
Init Redis
Init Cluster/Node/Instance
# init all redis instances on group <cluster>./redis.yml -l <cluster> # init redis cluster# init redis node./redis.yml -l 10.10.10.10 # init redis node# init one specific redis instance 10.10.10.11:6379./redis.yml -l 10.10.10.11 -e redis_port=6379 -t redis
Beware that redis cannot be reloaded online; you have to restart redis to make config effective.
Use Redis CLI
Access redis instance with redis-cli:
$ redis-cli -h 10.10.10.10 -p 6379# <--- connect with host and port10.10.10.10:6379> auth redis.ms # <--- auth with passwordOK
10.10.10.10:6379> set a 10# <--- set a keyOK
10.10.10.10:6379> get a # <--- get a key back"10"
Redis also has a redis-benchmark which can be used for benchmark and generate load on redis server:
# promote a redis instance to primary> REPLICAOF NO ONE
"OK"# make a redis instance replica of another instance> REPLICAOF 127.0.0.1 6799"OK"
HA with Sentinel
You have to enable HA for redis standalone m-s cluster manually with your redis sentinel.
Take the 4-node sandbox as an example, a redis sentinel cluster redis-meta is used to manage the redis-ms standalone cluster.
# for each sentinel, add redis master to the sentinel with:$ redis-cli -h 10.10.10.11 -p 26379 -a redis.meta
10.10.10.11:26379> SENTINEL MONITOR redis-ms 10.10.10.10 6379110.10.10.11:26379> SENTINEL SET redis-ms auth-pass redis.ms # if auth enabled, password has to be configured
If you wish to remove a redis master from sentinel, use SENTINEL REMOVE <name>.
You can configure multiple redis master on sentinel cluster with redis_sentinel_monitor.
redis_sentinel_monitor:# primary list for redis sentinel, use cls as name, primary ip:port- {name:redis-src, host:10.10.10.45, port:6379 ,password:redis.src, quorum:1}- {name:redis-dst, host:10.10.10.48, port:6379 ,password:redis.dst, quorum:1}
And refresh the master list on sentinel cluster with:
./redis.yml -l redis-meta -t redis-ha # replace redis-meta if your sentinel cluster has different name
use redis_clean = true and redis_safeguard = false to force clean redis data
This happens when you run redis.yml to init a redis instance that is already running, and redis_clean is set to false.
If redis_clean is set to true (and the redis_safeguard is set to false, too), the redis.yml playbook will remove the existing redis instance and re-init it as a new one, which makes the redis.yml playbook fully idempotent.
ABORT due to redis_safeguard enabled
This happens when removing a redis instance with redis_safeguard set to true.
You can disable redis_safeguard to remove the Redis instance. This is what redis_safeguard is for.
How to add a single new redis instance on this node?
Use bin/redis-add <ip> <port> to deploy a new redis instance on node.
How to remove a single redis instance from the node?
bin/redis-rm <ip> <port> to remove a single redis instance from node
17 - FERRET
Ferret, the mongo over postgres
MongoDB has lost its open-source appeal and is no longer suitable for many cases.
In contrast, PostgreSQL offers robust, native JSON support and outperforms MongoDB as a document database.
Thus, FerretDB provides a mongo wire-protocol-compatible layer upon postgres, enabling MongoDB users to migrate smoothly to PostgreSQL’s superior platform.
FERRET is an OPTIONAL module in Pigsty.
It requires the documentdb extensions to work since v2.0.
Pigsty has packaged that and provides a mongo.yml template to help you deploy a FerretDB cluster with ease.
Configuration
Configure ferret module, and use multiple ferret nodes.
You can access FerretDB using MongoDB connection strings with any MongoDB driver in any language. Here’s an example using the mongosh CLI tool:
$ mongosh
Current Mongosh Log ID: 67ba8c1fe551f042bf51e943
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.4.0
Using MongoDB: 7.0.77
Using Mongosh: 2.4.0
For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/
test>
mongosh 'mongodb://dbuser_meta:[email protected]:27017/meta'# Business admin usermongosh 'mongodb://dbuser_view:[email protected]:27017/meta'# Read-only user
Quick Start
You can connect to FerretDB and use it as if it were a MongoDB cluster.
Before deploying a Mongo (FerretDB) cluster, you need to define it in the inventory using the relevant parameters.
The following example uses the default single-node pg-meta cluster’s meta database as FerretDB’s underlying storage:
all:children:#----------------------------------## ferretdb for mongodb on postgresql#----------------------------------## ./mongo.yml -l ferretferret:hosts:10.10.10.10:{mongo_seq:1}vars:mongo_cluster:ferretmongo_pgurl:'postgres://mongod:[email protected]:5432/meta'
Here, mongo_cluster and mongo_seq are essential identity parameters. For FerretDB, mongo_pgurl is also required to specify the underlying PG location.
Note that the mongo_pgurl parameter requires a PostgreSQL superuser. In this example, a dedicated mongod superuser is defined for FerretDB.
Note that FerretDB’s authentication is entirely based on PostgreSQL. You can create other regular users using either FerretDB or PostgreSQL.
PostgreSQL Cluster
FerretDB 2.0+ requires an extension: DocumentDB, which depends on several other extensions. Here’s a template for creating a PostgreSQL cluster for FerretDB:
all:children:#----------------------------------## pgsql (singleton on current node)#----------------------------------## postgres cluster: pg-metapg-meta:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-metapg_users:- {name:mongod ,password:DBUser.Mongo ,pgbouncer:true ,roles:[dbrole_admin ] ,superuser:true ,comment:ferretdb super user }- {name:dbuser_meta ,password:DBUser.Meta ,pgbouncer:true ,roles:[dbrole_admin] ,comment:pigsty admin user }- {name:dbuser_view ,password:DBUser.Viewer ,pgbouncer:true ,roles:[dbrole_readonly] ,comment:read-only viewer for meta database }pg_databases:- {name:meta, owner:mongod ,baseline:cmdb.sql ,comment:pigsty meta database ,schemas:[pigsty] ,extensions:[documentdb, postgis, vector, pg_cron, rum ]}pg_hba_rules:- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}- {user:mongod , db:all ,addr:world ,auth:pwd ,title:'mongodb password access from everywhere'}pg_extensions:- documentdb, citus, postgis, pgvector, pg_cron, rumpg_parameters:cron.database_name:metapg_libs:'pg_documentdb, pg_documentdb_core, pg_cron, pg_stat_statements, auto_explain'# add timescaledb to shared_preload_libraries
High Availability
You can use Services to connect to a highly available PostgreSQL cluster and deploy multiple FerretDB instance replicas with L2 VIP binding for FerretDB layer high availability.
Docker is an optional module in Pigsty, it is downloaded by default but not installed.
You have to explicitly enable it before using.
Configuration
Configure docker registry, proxy, images, etc...
Parameters
Customize docker components with 8 parameters
Administration
Manage docker images, containers, etc...
Playbooks
Ansible playbooks that can be used in docker module
Monitoring
Dashboards, metrics, record & alerting rules.
FAQ
Frequently Asked Questions about docker module
18.1 - Configure
Configure your docker setup
Pigsty includes built-in Docker support, allowing you to quickly deploy containerized applications.
Quick Start
To install docker on nodes, set the docker_enabled parameter to true.
all:vars:infra:hosts:10.10.10.10:{infra_seq:1, nodename:infra-1 }10.10.10.11:{infra_seq:2, nodename:infra-2 }vars:docker_enabled:true# Install Docker on this group
Then run the docker.yml playbook (on target hosts/groups):
The images defined in docker_image will be pulled during the docker_image task, one by one.
And the local docker image cache with .tgz suffix matching the docker_image_cache glob list will be loaded into docker with docker load
docker_enabled:false# enable docker on this node?docker_data:/var/lib/docker # docker data directory, /var/lib/docker by defaultdocker_storage_driver:overlay2 # docker storage driver, can be zfs, btrfsdocker_cgroups_driver:systemd # docker cgroup fs driver: cgroupfs,systemddocker_registry_mirrors:[]# docker registry mirror listdocker_exporter_port:9323# docker metrics exporter port, 9323 by defaultdocker_image:[]# docker image to be pulled after bootstrapdocker_image_cache:/tmp/docker/*.tgz# docker image cache glob pattern
docker_enabled
name: docker_enabled, type: bool, level: G/C/I
enable docker on this node? default value is false
docker_data
name: docker_data, type: path, level: C
Docker data directory, /var/lib/docker by default.
all:vars:infra:hosts:10.10.10.10:{infra_seq:1, nodename:infra-1 }10.10.10.11:{infra_seq:2, nodename:infra-2 }vars:docker_enabled:true# Install Docker on this group
Then run the docker.yml playbook (on target hosts/groups):
./docker.yml -l infra
Docker will be installed on that infra group.
infra is a placeholder
We are using the infra group here as an example, you can define it elsewhere as long as it applies to expected hosts.
Repository
Docker repo is part of the infra repo module, and will be automatically added during repo build.
./node.yml -t node_repo -e node_repo_modules=infra -l infra
Upgrade
To upgrade Docker Daemon, using the ansible command, add docker repo, then:
~/pigsty
ansible infra -m package -b -a 'name=docker-ce state=latest'
It will upgrade the docker-ce package to the latest version available on your configured repositories.
Remove
To remove Docker Daemon, using the ansible command to run:
~/pigsty
ansible infra -m package -b -a 'name=docker-ce state=absent'
It will remove the docker-ce package with your os package manager.
Applications
Pigsty provides ready-to-use software templates based on Docker Compose to deploy external applications
seamlessly integrated with Pigsty-managed database clusters.
18.4 - Playbook
setup docker with playbook
The DOCKER module has only one playbook: docker.yml
to install docker daemon & docker compose on target node.
Run this playbook on any host will install docker-ce and docker-compose-plugin on target node with docker_enabled: true flag.
Here are the available subtasks in the docker.yml playbook:
docker_install: Install Docker and Docker Compose packages on the node.
docker_admin: Add specified users to the Docker administrator user group.
docker_config: Generate Docker daemon service configuration file.
docker_launch: Start the Docker daemon service.
docker_register: Register Docker daemon as a Prometheus monitoring target.
docker_image: Attempt to load prepackaged Docker images from /tmp/docker/*.tgz if present.
The Docker module does not provide a dedicated playbook for uninstalling Docker. If you need to uninstall Docker, you can manually stop the Docker service and then uninstall it:
systemctl stop docker # Stop Docker daemon serviceyum remove docker-ce docker-compose-plugin # Uninstall Docker on EL systemsapt remove docker-ce docker-compose-plugin # Uninstall Docker on Debian systems
18.5 - Monitor
docker monitoring and dashboards
Pigsty will add docker daemon to monitoring target if that node is docker_enabled = true
But there are no default dashboards and alerting rules for docker module, you can add your own rules to prometheus & grafana.
18.6 - FAQ
frequently asked questions
Who Can Run Docker Commands?
By default, Pigsty adds both the management user running the playbook on the remote host (i.e., the SSH login user) and the user defined by the node_admin_username parameter to the operating-system group docker.
Any account in this group can manage Docker via the docker CLI.
Need to grant another user Docker access? Just add that OS user to the docker group:
sudo usermod -aG docker <username>
Working Through a Proxy
During installation, if the proxy_env parameter is set, Pigsty writes the specified HTTP proxy settings to /etc/docker/daemon.json.
Docker will then route all image pulls from upstream registries through this proxy.
Tip: Running the configure playbook with the -x flag automatically captures your current shell’s proxy variables and injects them into proxy_env.
Using Mirror Registries
Inside mainland China you may encounter Great Firewall restrictions. Historically, mirrors such as quay.io could be used:
docker login quay.io # enter your credentials to log in
Update (June 2024): All previously accessible Docker mirrors in China have now been blocked. Please pull images via a proxy.
Adding Docker to Monitoring
When the Docker module is installed, you can register Docker as a Prometheus target for a specific node by running the docker_register (alias register_prometheus) task:
Pigsty ships a collection of software templates that launch popular stacks with Docker Compose—ready to use out of the box.
Just make sure the Docker module is installed first.
19 - APP
Run self-hosting application templates with pigsty and docker composes
Supabase
Self-Hosting Supabase
Odoo
Run Odoo the OSS ERP
Dify
Run Dify the AI workflow
pgAdmin
Run official Admin GUI Tool
19.1 - Playbook
Run docker compose application
Pigsty has built-in support for Docker and a series software that use PostgreSQL as primary storage.
You can run stateless application with docker compose, and store the data in the external HA PostgreSQL (Redis/MinIO/…) clusters.
There’s a dedicate playbook app.yml that can help your running docker compose app easily
19.2 - pgAdmin
Launch the official GUI Admin Tool for PostgreSQL
pgAdmin is the most popular and feature rich Open Source administration and development platform for PostgreSQL,
the most advanced Open Source database in the world.
Quick Start
Pigsty has built-in (but optional) support for pgAdmin, it launches pgadmin with Docker Compose:
./docker.yml
./app.yml -e app=pgadmin
The default port for pgadmin is 8885, and you can access it through IP:Port : http://10.10.10.10:8885.
The default credentials are defined in .env, username: [email protected], password: pigsty.
Customize
Customize the pgadmin configuration in /opt/pgadmin/.env and manage it with docker compose.
You can also customize the apps parameter and override default .env configuration with:
all:children:infra:hosts:10.10.10.10:{infra_seq:1}vars:docker_enabled:trueapp:pgadmin # specify app name (pgadmin) to be installed (in the apps)apps:# define all applicationssupabase:# the definition of pgadmin appconf:# override /opt/supabase/.envPGADMIN_DEFAULT_EMAIL:[email protected]PGADMIN_DEFAULT_PASSWORD:yourPasswordPGADMIN_LISTEN_ADDRESS:0.0.0.0PGADMIN_PORT:8885PGADMIN_SERVER_JSON_FILE:/pgadmin4/servers.jsonPGADMIN_REPLACE_SERVERS_ON_STARTUP:true
To launch the application, run:
./app.yml -l infra
Domain & Cert
To access pgadmin through nginx (rather than access port 8885 directly), configure the infra portal with:
Then run make nginx to update nginx config, and configure local static DNS record <your_ip_address> adm.pigsty in /etc/hosts or local / public DNS server.
Pigsty will issue self-signed SSL certs for the domains listed in infra_portal automatically.
If you want to use real domain names, define cerbot entry and run make cert, check SSL Certs for details.
all:vars:# make sure your domain name (adm.pigsty.cc) is resolved to your public IPcertbot_sign:true# sign real HTTPS certs with certbot (Internet access required!)infra_portal:pgadmin :{domain:adm.pigsty.cc ,endpoint:"127.0.0.1:8885", certbot:adm.pigsty.cc }
19.3 - Supabase
Self-host enterprise-grade Supabase with Pigsty, featuring monitoring, high availability, PITR, IaC, and 400+ PostgreSQL extensions.
Supabase is great, but having your own Supabase is even better.
Pigsty helps you build enterprise-grade Supabase on your own servers (physical/virtual machines/cloud servers) with one-click deployment —
more extensions, better performance, deeper control, and much more cost-effective.
Pigsty is one of the three 3rd party self-hosting tutorials listed in the official Supabase docs
Quick Start
Prepare a Linux server, follow the Pigsty standard installation process, select the supabase configuration template, and execute the following commands:
Supabase is a BaaS (Backend as Service), an open-source Firebase alternative, and the most popular database + backend solution in the AI Agent era.
Supabase wraps PostgreSQL and provides authentication, messaging, edge functions, object storage, and automatically generates REST API and GraphQL API based on PostgreSQL database schemas.
Supabase aims to provide developers with a one-stop backend solution, reducing the complexity of developing and maintaining backend infrastructure.
It allows developers to eliminate most backend development work — developers only need to understand database design and frontend to quickly deliver applications!
Developers can quickly complete a full application with just frontend development and database schema design using Vibe Coding.
Currently, Supabase is the most popular open-source project in the PostgreSQL open-source ecosystem, with 80,000 stars on GitHub.
Supabase also provides “generous” free cloud service quotas for small entrepreneurs — 500 MB of free space, which is sufficient for storing user tables, view counts, and similar data.
Why Self-Host?
Since Supabase cloud service is so attractive, why self-host?
The most intuitive reason is what we mentioned in “Are Cloud Databases an Intelligence Tax?”: when your data/computing scale exceeds the cloud computing applicable spectrum (Supabase: 4C/8G/500MB free storage), costs can easily explode.
Moreover, currently, sufficiently reliable local enterprise-grade NVMe SSDs have a three to four order of magnitude advantage in cost-effectiveness compared to cloud storage, and self-hosting can better leverage this advantage.
Another important reason is functionality — Supabase cloud service functionality is limited. Many powerful PostgreSQL extensions cannot be provided as cloud services due to multi-tenant security challenges and licensing issues.
Therefore, although extensions are PostgreSQL’s core feature, only 64 extensions are available on Supabase cloud service.
Self-built Supabase with Pigsty provides up to 437 ready-to-use PostgreSQL extensions.
Additionally, autonomy and avoiding vendor lock-in are important reasons for self-hosting — although Supabase aims to provide an open-source alternative to Google Firebase without vendor lock-in, the threshold for self-building enterprise-grade Supabase to high standards is actually quite high.
Supabase includes a series of PostgreSQL extension plugins developed and maintained by them, and plans to replace the native PostgreSQL kernel with the acquired OrioleDB, but these kernels and extensions are not provided in the official PGDG repository.
This is actually a form of implicit vendor lock-in, preventing users from self-building using methods other than the supabase/postgres Docker image. Pigsty provides an open-source, transparent, and universal solution to solve this problem.
We package all 10 missing extensions developed and used by Supabase into ready-to-use RPM/DEB packages, ensuring they are available on all mainstream Linux operating system distributions:
Filter and block specific query statements using execution plan costs, C extension, provided by PIGSTY
Meanwhile, we install most extensions by default in Supabase self-hosting deployment. You can refer to the available extension list to enable them as needed.
Additionally, Pigsty handles the automatic setup of underlying high availabilityPostgreSQL database clusters, high availability MinIO object storage clusters, and even Docker container infrastructure deployment and Nginx reverse proxy, domain configuration and HTTPS certificate issuance. You can deploy any number of stateless Supabase container clusters using Docker Compose and store state in external Pigsty self-hosted database services.
In this self-hosting deployment architecture, you gain the freedom to use different kernels (PostgreSQL 15-18, OrioleDB), the freedom to install 437 extensions, the freedom to scale Supabase/Postgres/MinIO,
the freedom from database operational chores, and the freedom from vendor lock-in to run locally indefinitely. Compared to the cost of using cloud services, the price is just preparing servers and typing a few more commands.
Single Node Quick Start
Let’s start with single-node Supabase deployment. We’ll introduce multi-node high availability deployment methods later.
Before deploying Supabase, please modify the parameters (domain and passwords) in the automatically generated pigsty.yml configuration file according to your actual situation.
If it’s just local development testing, you can skip this for now. We’ll introduce how to further customize through configuration file modifications later.
If configured correctly, after about ten minutes, you can access the Supabase Studio graphical management interface locally via http://<your_ip_address>:8000.
The default username and password are: supabase and pigsty.
DockerHub blocked in mainland China
In mainland China, Pigsty uses DockerHub mirror sites provided by 1Panel and 1ms to download Supabase-related images by default, which may be slow.
You can also configure proxy and mirror sites yourself, or manually pull images with cd /opt/supabase; docker compose pull.
We also provide Supabase self-hosting expert consulting services including complete offline installation solutions.
Using Supabase object storage requires HTTPS/domain
If you need to use object storage functionality, you need to access Supabase via domain and HTTPS, otherwise errors will occur.
Please change passwords for production deployment!
For serious production deployments, must change all default passwords!
Key Technical Decisions for Self-Hosting
Here are some key technical decisions involved in self-hosting Supabase for your reference:
Using the default single-node deployment, Supabase cannot enjoy PostgreSQL/MinIO high availability capabilities.
Nevertheless, single-node deployment still has significant advantages compared to the official pure Docker Compose solution: for example, out-of-the-box monitoring systems, the ability to freely install extensions, component scaling capabilities, and providing fallback database point-in-time recovery capabilities.
If you only have one server or choose to self-host on cloud servers, Pigsty recommends using external S3 instead of local MinIO as object storage to store PostgreSQL backups and support Supabase Storage services.
Such deployment can provide a fallback-level RTO (hour-level recovery time)/RPO (MB-level data loss) disaster recovery level under single-machine deployment conditions during failures.
In serious production deployments, Pigsty recommends using at least 3-4 node deployment strategies to ensure both MinIO and PostgreSQL use multi-node deployments that meet enterprise-grade high availability requirements. In this case, you need to prepare more nodes and disks accordingly and adjust cluster configurations in the pigsty.yml configuration manifest, as well as access information in supabase cluster configuration to use high availability access points.
Some Supabase functionality requires sending emails, so SMTP services are needed. Unless purely for internal networks, for serious production deployments, using SMTP cloud services is recommended. Self-built email servers easily have their emails marked as spam and rejected.
If your service is directly exposed to the public network, we strongly recommend using real domains and HTTPS certificates and accessing through Nginx Portal.
Next, we’ll discuss some advanced topics in sequence: how to further improve Supabase security, availability, and performance based on single-node deployment.
Advanced Topic: Security Hardening
Pigsty Base Components
For serious production deployments, we strongly recommend changing Pigsty default passwords.
Because these default values are public and well-known, going to production without changing passwords is like streaking:
Generate a JWT_SECRET longer than 40 characters and use the tools in the tutorial to sign ANON_KEY and SERVICE_ROLE_KEY JWTs.
Use the tools provided in the tutorial to generate an ANON_KEY JWT based on JWT_SECRET and expiration time attributes. This is the credential for anonymous users.
Use the tools provided in the tutorial to generate a SERVICE_ROLE_KEY based on JWT_SECRET and expiration time attributes. This is the credential for higher-privilege service roles.
Setup PG_META_CRYPTO_KEY with a random string at least 32 char long for securing connection strings between Studio and postgres-meta
If your PostgreSQL business user uses a password different from the default, please modify the POSTGRES_PASSWORD value accordingly
If your object storage uses a password different from the default, please modify the S3_ACCESS_KEY and S3_SECRET_KEY values accordingly
After modifying Supabase credentials, you can restart Docker Compose containers to apply the new configuration:
./app.yml -t app_config,app_launch
cd /opt/supabase; make up
Advanced Topic: Domain Integration
If you’re using Supabase on localhost or within a LAN, you can choose IP:Port direct connection to Kong’s exposed HTTP port 8000 to access Supabase.
You can use an internal static DNS domain, but for serious production deployments, we recommend using real domain + HTTPS to access Supabase.
In this case, your server should have a public IP address, you should own a domain, use DNS resolution services provided by cloud/DNS/CDN providers to point it to the installation node’s public IP (optional fallback: local /etc/hosts static resolution).
A simple approach is to batch replace the placeholder domain (supa.pigsty) with your actual domain, say supa.pigsty.cc:
sed -ie 's/supa.pigsty/supa.pigsty/g' ~/pigsty/pigsty.yml
If you haven’t configured it beforehand, reload Nginx and Supabase configurations:
make nginx # Reload nginx configurationmake cert # Apply for free HTTPS certificate with certbot./app.yml # Reload Supabase configuration
The modified configuration should look like the following snippet:
all:vars:infra_portal:supa :domain:supa.pigsty.cc # Replace with your domain!endpoint:"10.10.10.10:8000"websocket:truecertbot:supa.pigsty.cc # Certificate name, usually same as domainchildren:supabase:vars:supabase:# the definition of supabase appconf:# override /opt/supabase/.envSITE_URL:https://supa.pigsty # <------- Change This to your external domain nameAPI_EXTERNAL_URL:https://supa.pigsty # <------- Otherwise the storage api may not work!SUPABASE_PUBLIC_URL:https://supa.pigsty # <------- DO NOT FORGET TO PUT IT IN infra_portal!
Complete domain/HTTPS configuration can refer to the Certificate Management tutorial. You can also use Pigsty’s built-in local static resolution and self-signed HTTPS certificates as fallback.
Advanced Topic: External Object Storage
You can use S3 or S3-compatible services as object storage for PostgreSQL backups and Supabase usage. Here we use Alibaba Cloud OSS object storage as an example.
Pigsty provides a terraform/spec/aliyun-meta-s3.tf template
that can be used to deploy a server and an OSS bucket on Alibaba Cloud.
First, modify the S3-related configuration in all.children.supa.vars.apps.[supabase].conf, pointing it to the Alibaba Cloud OSS bucket:
# if using s3/minio as file storageS3_BUCKET:data # Replace with S3-compatible service connection informationS3_ENDPOINT:https://sss.pigsty:9000 # Replace with S3-compatible service connection informationS3_ACCESS_KEY:s3user_data # Replace with S3-compatible service connection informationS3_SECRET_KEY:S3User.Data # Replace with S3-compatible service connection informationS3_FORCE_PATH_STYLE:true# Replace with S3-compatible service connection informationS3_REGION:stub # Replace with S3-compatible service connection informationS3_PROTOCOL:https # Replace with S3-compatible service connection information
Reload Supabase configuration with the following command:
./app.yml -t app_config,app_launch
You can also use S3 as PostgreSQL backup repository by adding an aliyun backup repository definition in all.vars.pgbackrest_repo:
all:vars:pgbackrest_method:aliyun # pgbackrest backup method: local,minio,[other user-defined repositories...], in this example backup is stored to MinIOpgbackrest_repo:# pgbackrest backup repository: https://pgbackrest.org/configuration.html#section-repositoryaliyun:# Define a new backup repository aliyuntype:s3 # Alibaba Cloud OSS is S3-compatible object storages3_endpoint:oss-cn-beijing-internal.aliyuncs.coms3_region:oss-cn-beijings3_bucket:pigsty-osss3_key:xxxxxxxxxxxxxxs3_key_secret:xxxxxxxxs3_uri_style:hostpath:/pgbackrestbundle:y# bundle small files into a single filebundle_limit:20MiB # Limit for file bundles, 20MiB for object storagebundle_size:128MiB # Target size for file bundles, 128MiB for object storagecipher_type:aes-256-cbc # enable AES encryption for remote backup repocipher_pass:pgBackRest.MyPass # Set an encryption password, pgBackrest backup repository encryption passwordretention_full_type:time # retention full backup by time on minio reporetention_full:14# keep full backup for the last 14 days
Then specify using the aliyun backup repository in all.vars.pgbackrest_method and reset pgBackrest backup:
./pgsql.yml -t pgbackrest
Pigsty will switch the backup repository to external object storage. More backup configurations can refer to PostgreSQL Backup documentation.
Advanced Topic: Using SMTP
You can use SMTP to send emails by modifying the supabase application configuration and adding SMTP information:
all:children:supabase:# supa groupvars:# supa group varsapps:# supa group app listsupabase:# the supabase appconf:# the supabase app conf entriesSMTP_HOST:smtpdm.aliyun.com:80SMTP_PORT:80SMTP_USER:[email protected]SMTP_PASS:your_email_user_passwordSMTP_SENDER_NAME:MySupabaseSMTP_ADMIN_EMAIL:[email protected]ENABLE_ANONYMOUS_USERS:false
Don’t forget to use app.yml to reload the configuration
Advanced Topic: True High Availability
After these configurations, you have an enterprise-grade Supabase (basic single-machine version) with public domain, HTTPS certificate, SMTP, PITR backup, monitoring, IaC, and 400+ extensions.
For high availability configuration, please refer to other parts of Pigsty documentation. If you’re too lazy to read and learn, we provide hands-on Supabase self-hosting expert consulting services — ¥2000 to save you from the hassle of tinkering and downloading.
Single-node RTO/RPO relies on external object storage services for fallback. If your node fails, backups are retained in external S3 storage, and you can redeploy Supabase on a new node and restore from backup.
Such deployment can provide a minimum standard RTO (hour-level recovery time)/RPO (MB-level data loss) fallback disaster recovery level during failures.
To achieve RTO < 30s with zero data loss failover, you need to use multi-node high availability deployment, which involves:
ETCD: DCS needs three or more nodes to tolerate one node failure.
PGSQL: PostgreSQL synchronous commit mode without data loss, recommend using at least three nodes.
INFRA: Monitoring infrastructure failure has less impact, recommend using dual replicas in production
Supabase stateless containers themselves can also be multi-node replicas to achieve high availability.
In this case, you also need to modify PostgreSQL and MinIO access points to use DNS/L2 VIP/HAProxy and other high availability access points
For these parts, you only need to refer to the documentation of each module in Pigsty for configuration and deployment.
We recommend referring to the configurations in conf/ha/trio.yml and conf/ha/safe.yml to upgrade cluster scale to three nodes or more.
19.4 - Odoo
Self-Hosting Odoo, the Open Source ERP
Odoo is an open-source enterprise resource planning (ERP) software
that provides a full suite of business applications, including CRM, sales, purchasing, inventory, production, accounting,
and other management functions. Odoo is a typical web application that uses PostgreSQL as the underlying database.
All your business on one platform, Simple, efficient, yet affordable
The latest version of Odoo is 19.0
Quick Start
curl -fsSL https://repo.pigsty.io/get | bash -s v3.7.0;cd ~/pigsty
./bootstrap # install ansible./configure -c app/odoo # use odoo config (please CHANGE CREDENTIALS in pigsty.yml)./install.yml # install pigsty./docker.yml # install docker compose./app.yml # launch odoo stateless part with docker
The default credentials are admin : admin
Config Template
The conf/app/odoo.yml defines a template config file
that defines the resources required for a single Odoo instance.
all:children:# the odoo application (default username & password: admin/admin)odoo:hosts:{10.10.10.10:{}}vars:app:odoo # specify app name to be installed (in the apps)apps:# define all applicationsodoo:# app name should have the corresponding ~/app/odoo folderfile:# optional directory to be created- {path:/data/odoo ,state:directory, owner:100, group:101}- {path:/data/odoo/webdata ,state:directory, owner:100, group:101}- {path:/data/odoo/addons ,state:directory, owner:100, group:101}conf:# override /opt/<app>/.env config filePG_HOST:10.10.10.10# postgres hostPG_PORT:5432# postgres portPG_USERNAME:odoo # postgres userPG_PASSWORD:DBUser.Odoo # postgres passwordODOO_PORT:8069# odoo app portODOO_DATA:/data/odoo/webdata # odoo webdataODOO_ADDONS:/data/odoo/addons # odoo pluginsODOO_DBNAME:odoo # odoo database nameODOO_VERSION:19.0# odoo image version# the odoo databasepg-odoo:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-odoopg_users:- {name:odoo ,password:DBUser.Odoo ,pgbouncer:true ,roles:[ dbrole_admin ] ,createdb:true ,comment:admin user for odoo service }- {name:odoo_ro ,password:DBUser.Odoo ,pgbouncer:true ,roles:[ dbrole_readonly ] ,comment:read only user for odoo service }- {name:odoo_rw ,password:DBUser.Odoo ,pgbouncer:true ,roles:[ dbrole_readwrite ] ,comment:read write user for odoo service }pg_databases:- {name:odoo ,owner:odoo ,revokeconn:true ,comment:odoo main database }pg_hba_rules:- {user:all ,db:all ,addr:172.17.0.0/16 ,auth:pwd ,title:'allow access from local docker network'}- {user:dbuser_view , db:all ,addr:infra ,auth:pwd ,title:'allow grafana dashboard access cmdb from infra nodes'}infra:{hosts:{10.10.10.10:{infra_seq:1}}}etcd:{hosts:{10.10.10.10:{etcd_seq:1 } }, vars:{etcd_cluster:etcd } }#minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }vars:# global variablesversion:v3.7.0 # pigsty version stringadmin_ip:10.10.10.10# admin node ip addressregion:default # upstream mirror region: default|china|europenode_tune:oltp # node tuning specs: oltp,olap,tiny,critpg_conf:oltp.yml # pgsql tuning specs: {oltp,olap,tiny,crit}.ymldocker_enabled:true# enable docker on app group#docker_registry_mirrors: ["https://docker.m.daocloud.io"] # use dao cloud mirror in mainland chinaproxy_env:# global proxy env when downloading packages & pull docker imagesno_proxy:"localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.tsinghua.edu.cn"#http_proxy: 127.0.0.1:12345 # add your proxy env here for downloading packages or pull images#https_proxy: 127.0.0.1:12345 # usually the proxy is format as http://user:[email protected]#all_proxy: 127.0.0.1:12345infra_portal:# domain names and upstream servershome :{domain:h.pigsty }grafana :{domain:g.pigsty ,endpoint:"${admin_ip}:3000", websocket:true}prometheus :{domain:p.pigsty ,endpoint:"${admin_ip}:9058"}alertmanager :{domain:a.pigsty ,endpoint:"${admin_ip}:9059"}blackbox :{endpoint:"${admin_ip}:9115"}loki :{endpoint:"${admin_ip}:3100"}minio :{domain:m.pigsty ,endpoint:"${admin_ip}:9001",scheme:https ,websocket:true}odoo :{domain:odoo.pigsty, endpoint:"127.0.0.1:8069",websocket:true}#cert: /path/to/crt ,key: /path/to/key# setup your own domain name here ^^^, or use default domain name, or ip + 8069 port direct access# certbot --nginx --agree-tos --email [email protected] -n -d odoo.your.domain # replace with your email & odoo domain#----------------------------------## Credential: CHANGE THESE PASSWORDS#----------------------------------##grafana_admin_username: admingrafana_admin_password:pigsty#pg_admin_username: dbuser_dbapg_admin_password:DBUser.DBA#pg_monitor_username: dbuser_monitorpg_monitor_password:DBUser.Monitor#pg_replication_username: replicatorpg_replication_password:DBUser.Replicator#patroni_username: postgrespatroni_password:Patroni.API#haproxy_admin_username: adminhaproxy_admin_password:pigstyrepo_modules:infra,node,pgsql,dockerrepo_packages:[node-bootstrap, infra-package, infra-addons, node-package1, node-package2, pgsql-utility, docker ]repo_extra_packages:[pg18-main ]pg_version:18
Basics
Check .env file for configurable environment variables:
make up # pull up odoo with docker compose in minimal modemake run # launch odoo with docker , local data dir and external PostgreSQLmake view # print odoo access pointmake log # tail -f odoo logsmake info # introspect odoo with jqmake stop # stop odoo containermake clean # remove odoo containermake pull # pull latest odoo imagemake rmi # remove odoo imagemake save # save odoo image to /tmp/docker/odoo.tgzmake load # load odoo image from /tmp/docker/odoo.tgz
Use External PostgreSQL
You can use external PostgreSQL for Odoo. Odoo will create its own database during setup, so you don’t need to do that
pg_users:[{name:dbuser_odoo ,password:DBUser.Odoo ,pgbouncer:true ,roles:[ dbrole_admin ] ,comment:admin user for odoo database } ]pg_databases:[{name:odoo ,owner:dbuser_odoo ,revokeconn:true ,comment:odoo primary database } ]
And create business user & database with:
bin/pgsql-user pg-meta dbuser_odoo
#bin/pgsql-db pg-meta odoo # odoo will create the database during setup
If you want to access odoo through SSL, you have to trust files/pki/ca/ca.crt on your browser (or use the dirty hack thisisunsafe in chrome)
Feedback
19.5 - Dify
setup docker with playbook
Dify is a generative AI application innovation engine and open-source LLM application development platform.
It provides capabilities from Agent construction to AI workflow orchestration, RAG retrieval, and model management,
helping users easily build and operate generative AI-native applications.
Pigsty provides support for self-hosting Dify, allowing you to deploy Dify with a single command while storing critical state in externally managed PostgreSQL.
You can use pgvector in the same PostgreSQL instance as a vector database, further simplifying deployment.
Dify listens on port 5001 by default. You can access it via browser at http://<ip>:5001 and set up your initial user credentials to log in.
After Dify starts, you can install various extensions, configure system models, and begin using it!
Why Self-Host
There are many reasons to self-host Dify, but the primary motivation is data security.
The DockerCompose template provided by Dify uses basic default database images, lacking enterprise-grade features
like high availability, disaster recovery, monitoring, IaC, and PITR capabilities.
Pigsty elegantly solves these issues for Dify, deploying all components with a single command based on configuration files,
and using mirrors to resolve China region access challenges. This makes Dify deployment and delivery incredibly smooth.
It handles PostgreSQL master database, PGVector vector database, MinIO object storage, Redis, Prometheus monitoring,
Grafana visualization, Nginx reverse proxy, and free HTTPS certificates in one go.
Pigsty ensures all Dify state is stored in externally managed services, including metadata in PostgreSQL and other data in the filesystem.
Therefore, the Dify instance launched via Docker Compose becomes a stateless application that can be destroyed and rebuilt at any time, greatly simplifying operations.
Installation
Let’s start with single-node Dify deployment. We’ll cover production high-availability deployment methods later.
curl -fsSL https://repo.pigsty.cc/get | bash -s v3.7.0;cd ~/pigsty
./bootstrap # Prepare Pigsty dependencies./configure -c app/dify # Use Dify application templatevi pigsty.yml # Edit config file, modify domain and passwords./install.yml # Install Pigsty and various databases
When you use the ./configure -c app/dify command, Pigsty automatically generates the configuration file based on the conf/app/dify.yml
template and your current environment. You should modify passwords, domain, and other relevant parameters in the generated pigsty.yml configuration file
according to your actual needs, then use ./install.yml to execute the standard installation process.
Next, run docker.yml to install Docker and Docker Compose,
then use app.yml to complete Dify deployment:
./docker.yml # Install Docker and Docker Compose./app.yml # Deploy Dify stateless components using Docker
You can access the Dify Web management interface at http://<your_ip_address>:5001 on your local network.
Default username, email, and password will be prompted for setup on first login.
You can also use the locally resolved placeholder domain dify.pigsty, or follow the configuration below to use a real domain with HTTPS certificates.
Configuration
When you use the ./configure -c app/dify command for configuration,
Pigsty automatically generates the configuration file based on the conf/app/dify.yml template and your current environment.
Here’s a detailed explanation of the default configuration:
all:children:# the dify applicationdify:hosts:{10.10.10.10:{}}vars:app:dify # specify app name to be installed (in the apps)apps:# define all applicationsdify:# app name, should have corresponding ~/pigsty/app/dify folderfile:# data directory to be created- {path:/data/dify ,state:directory ,mode:0755}conf:# override /opt/dify/.env config file# change domain, mirror, proxy, secret keyNGINX_SERVER_NAME:dify.pigsty# A secret key for signing and encryption, gen with `openssl rand -base64 42` (CHANGE PASSWORD!)SECRET_KEY:sk-9f73s3ljTXVcMT3Blb3ljTqtsKiGHXVcMT3BlbkFJLK7U# expose DIFY nginx service with port 5001 by defaultDIFY_PORT:5001# where to store dify files? the default is ./volume, we'll use another volume created aboveDIFY_DATA:/data/dify# proxy and mirror settings#PIP_MIRROR_URL: https://pypi.tuna.tsinghua.edu.cn/simple#SANDBOX_HTTP_PROXY: http://10.10.10.10:12345#SANDBOX_HTTPS_PROXY: http://10.10.10.10:12345# database credentialsDB_USERNAME:difyDB_PASSWORD:difyai123456DB_HOST:10.10.10.10DB_PORT:5432DB_DATABASE:difyVECTOR_STORE:pgvectorPGVECTOR_HOST:10.10.10.10PGVECTOR_PORT:5432PGVECTOR_USER:difyPGVECTOR_PASSWORD:difyai123456PGVECTOR_DATABASE:difyPGVECTOR_MIN_CONNECTION:2PGVECTOR_MAX_CONNECTION:10pg-meta:hosts:{10.10.10.10:{pg_seq:1, pg_role:primary } }vars:pg_cluster:pg-metapg_users:- {name:dify ,password:difyai123456 ,pgbouncer:true ,roles:[ dbrole_admin ] ,superuser:true ,comment:dify superuser }pg_databases:- {name:dify ,owner:dify ,revokeconn:true ,comment:dify main database }pg_hba_rules:- {user:dify ,db:all ,addr:172.17.0.0/16 ,auth:pwd ,title:'allow dify access from local docker network'}node_crontab:['00 01 * * * postgres /pg/bin/pg-backup full']# make a full backup every 1aminfra:{hosts:{10.10.10.10:{infra_seq:1}}}etcd:{hosts:{10.10.10.10:{etcd_seq:1 } }, vars:{etcd_cluster:etcd } }#minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }vars:# global variablesversion:v3.7.0 # pigsty version stringadmin_ip:10.10.10.10# admin node ip addressregion:default # upstream mirror region: default|china|europenode_tune:oltp # node tuning specs: oltp,olap,tiny,critpg_conf:oltp.yml # pgsql tuning specs: {oltp,olap,tiny,crit}.ymldocker_enabled:true# enable docker on app group#docker_registry_mirrors: ["https://docker.1ms.run"] # use mirror in mainland chinaproxy_env:# global proxy env when downloading packages & pull docker imagesno_proxy:"localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.tsinghua.edu.cn"#http_proxy: 127.0.0.1:12345 # add your proxy env here for downloading packages or pull images#https_proxy: 127.0.0.1:12345 # usually the proxy is format as http://user:[email protected]#all_proxy: 127.0.0.1:12345infra_portal:# domain names and upstream servershome :{domain:h.pigsty }grafana :{domain:g.pigsty ,endpoint:"${admin_ip}:3000", websocket:true}prometheus :{domain:p.pigsty ,endpoint:"${admin_ip}:9058"}alertmanager :{domain:a.pigsty ,endpoint:"${admin_ip}:9059"}blackbox :{endpoint:"${admin_ip}:9115"}loki :{endpoint:"${admin_ip}:3100"}#minio : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }dify:# nginx server config for difydomain:dify.pigsty # REPLACE WITH YOUR OWN DOMAIN!endpoint:"10.10.10.10:5001"# dify service endpoint: IP:PORTwebsocket:true# add websocket supportcertbot:dify.pigsty # certbot cert name, apply with `make cert`#----------------------------------## Credential: CHANGE THESE PASSWORDS#----------------------------------##grafana_admin_username: admingrafana_admin_password:pigsty#pg_admin_username: dbuser_dbapg_admin_password:DBUser.DBA#pg_monitor_username: dbuser_monitorpg_monitor_password:DBUser.Monitor#pg_replication_username: replicatorpg_replication_password:DBUser.Replicator#patroni_username: postgrespatroni_password:Patroni.API#haproxy_admin_username: adminhaproxy_admin_password:pigsty#minio_access_key: minioadminminio_secret_key:minioadmin # minio root secret key, `minioadmin` by defaultrepo_extra_packages:[pg17-main ]pg_version:17
Checklist
Here’s a checklist of configuration items you need to focus on:
Best to specify an email address certbot_email for receiving certificate expiration notifications
Configure Dify’s NGINX_SERVER_NAME parameter to specify your actual domain
all:children:# Cluster definitiondify:# Dify groupvars:# Dify group variablesapps:# Application configurationdify:# Dify application definitionconf:# Dify application configurationNGINX_SERVER_NAME:dify.pigstyvars:# Global parameters#certbot_sign: true # Use Certbot to apply for free HTTPS certificatecertbot_email:[email protected]# Email for certificate application, used for expiration notifications, optionalinfra_portal:# Configure Nginx serverdify:# Dify server definitiondomain:dify.pigsty # Please replace with your own domain here!endpoint:"10.10.10.10:5001"# Please specify Dify's IP and port here (default auto-configured)websocket:true# Dify needs websocket enabledcertbot:dify.pigsty # Specify Certbot certificate name
Use the following command to apply for Nginx certificates:
# Apply for certificates, can also manually execute /etc/nginx/sign-cert scriptmake cert
# The above Makefile shortcut command actually executes the following playbook tasks:./infra.yml -t nginx_certbot,nginx_reload -e certbot_sign=true
Execute app.yml playbook to redeploy Dify service to make NGINX_SERVER_NAME configuration take effect.
./app.yml
File Backup
You can use restic to back up Dify’s filesystem. Dify’s data files are in the /data/dify directory. You can use the following commands to backup:
