Skip to content

Multi-Node

how to install pigsty on multiple nodes
how to install pigsty on multiple nodes

There is a configuration Tutorial for expanding Pigsty from one node to multiple nodes. While the easiest way is always pre-planing everything and provisioning them in one-pass.

1-node Setup

We already illustrated the 1-node installation in the Quick Start section, which may be the simplest setup.

ID IP Address NODE PGSQL INFRA ETCD
1 10.10.10.10 meta pg-meta-1 infra-1 etcd-1

It is not advised to put all the eggs in one basket, but even this one-node setup can be used for production, as long as an external MinIO / S3 / NFS… is configured for PG as remote backup repo.

There are lots of 1-node config templates for your reference. And provision VM with Vagrant meta.rb or Terraform meta.tf.

2-node Setup

The semi-HA setup

A two-node setup enables database replication and semi-HA capabilities:

ID IP Address NODE PGSQL INFRA ETCD
1 10.10.10.10 meta pg-meta-1 infra-1 etcd-1
2 10.10.10.11 node-1 pg-meta-2

While more robust than a single node setup, HA has limitations:

  • No automatic failover if node-1 fails - manual promotion of node-2 required
  • Automatic failover works if node-2 fails - node-1 gets promoted automatically

This “semi-HA” setup can only auto-recover from specific node failures.

You can use the dual.yml config template, and provision required VM with: Vagrant dual.rb to provision this environment.

3-node Setup

The true HA

A true HA setup that can automatically recover from any single node failure:

ID IP Address NODE PGSQL INFRA ETCD
1 10.10.10.10 node-1 pg-meta-1 infra-1 etcd-1
2 10.10.10.11 node-2 pg-meta-2 infra-2 etcd-2
3 10.10.10.12 node-3 pg-meta-3 infra-3 etcd-3

You can use the trio.yml config template, and provision required VM with: Vagrant trio.rb to provision this environment.

4-node Sandbox

This is a sandbox demo environment used in Pigsty, which has one infra node and three extra data nodes:

ID IP Address NODE PGSQL INFRA ETCD MINIO
1 10.10.10.10 meta pg-meta-1 infra-1 etcd-1 minio-1
2 10.10.10.11 node-1 pg-test-1
3 10.10.10.12 node-2 pg-test-1
4 10.10.10.13 node-3 pg-test-1

You can use the full.yml config template, and provision required VM with: Vagrant full.rb or Terraform full.tf to provision this environment.

5-node Building

This pro.yml is a 5-node building environment which contains supported Linux distros.

ID IP Address NODE PGSQL INFRA ETCD
1 10.10.10.8 el8 el8-1 infra-1
2 10.10.10.9 el9 el9-1 infra-2 etcd-2
3 10.10.10.12 u12 u12-1 infra-3
4 10.10.10.22 u22 u22-1 infra-4
5 10.10.10.24 u24 u24-1 infra-5

You can use the Vagrant pro.rb or Terraform pro.tf to provision this environment.

36-node Simulation

A production simulation environment (simu.yml) with 36 nodes covering all Pigsty components

IP Address SPEC NODE PGSQL INFRA ETCD MINIO REDIS
10.10.10.10 8C32G meta1 pg-meta-1 infra-1
10.10.10.11 8C32G meta2 pg-meta-2 infra-2
10.10.10.12 2C4G pg12 pg-v12-1
10.10.10.13 2C4G pg13 pg-v13-1
10.10.10.14 2C4G pg14 pg-v14-1
10.10.10.15 2C4G pg15 pg-v15-1
10.10.10.16 2C4G pg16 pg-v16-1
10.10.10.17 2C4G pg17 pg-v17-1
10.10.10.18 2C4G proxy1
10.10.10.19 2C4G proxy2
10.10.10.21 2C4G minio1 etcd-1 minio-1 redis-meta-1
10.10.10.22 2C4G minio2 etcd-2 minio-2 redis-meta-2
10.10.10.23 2C4G minio3 etcd-3 minio-3 redis-meta-3
10.10.10.24 2C4G minio4 etcd-4 minio-4 redis-meta-4
10.10.10.25 2C4G minio5 etcd-5 minio-5 redis-meta-5
10.10.10.40 1C2G node40 pg-pitr-1
10.10.10.41 1C2G node41 pg-test-1 redis-test-1
10.10.10.42 1C2G node42 pg-test-2 redis-test-2
10.10.10.43 1C2G node43 pg-test-3 redis-test-3
10.10.10.44 1C2G node44 pg-test-4 redis-test-4
10.10.10.45 1C2G node45 pg-src-1 redis-src-1
10.10.10.46 1C2G node46 pg-src-2 redis-src-2
10.10.10.47 1C2G node47 pg-src-3 redis-src-3
10.10.10.48 1C2G node48 pg-dst-1 redis-dst-1
10.10.10.49 1C2G node49 pg-dst-2 redis-dst-2
10.10.10.50 1C2G node50 pg-citus0-1
10.10.10.51 1C2G node51 pg-citus0-2
10.10.10.52 1C2G node52 pg-citus1-1
10.10.10.53 1C2G node53 pg-citus1-2
10.10.10.54 1C2G node54 pg-citus2-1
10.10.10.55 1C2G node55 pg-citus2-2
10.10.10.56 1C2G node56 pg-citus3-1
10.10.10.57 1C2G node57 pg-citus3-2
10.10.10.58 1C2G node58 pg-citus4-1
10.10.10.59 1C2G node59 pg-citus4-2
10.10.10.88 4C8G test

You can use the Vagrant simu.rb to provision this environment. You can run the entire simulation on a real server (72C / 256G) with libvirt as VM provider with vagrant.

  • 2 infra nodes, monitoring each other
  • 2 dedicated proxy nodes that run haproxy
  • 5-node etcd cluster which tolerates 2 node failures, and 5-node redis sentinel cluster
  • 5-node minio cluster with 4 disks on each node
  • 10 postgres clusters, pg13 - pg15, pg-src, pg-dst, pg-pitr, pg-test
  • 10-node citus cluster with 5 shards
  • redis standalone cluster redis-src and redis-dst, and native cluster redis-test